[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:32 dist-test-slave-cffh krb5kdc[1218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:32 dist-test-slave-cffh krb5kdc[1218](info): set up 2 sockets
Sep 27 10:54:32 dist-test-slave-cffh krb5kdc[1218](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 10:54:33.787429 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:33 dist-test-slave-cffh krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:33.907904 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36679
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:36679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:34.013438 1231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:34.013695 1231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:34.013736 1231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:34.017727 1231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:34.017786 1231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:34.017807 1231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:34.017827 1231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:34.017843 1231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:34.022854 1231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:36679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:34.023759 1231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:34.024793 1231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:54:34.033278 1231 server_base.cc:1034] running on GCE node
W20260927 10:54:34.033354 1237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:34.033354 1236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:34.033476 1239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:34.033998 1231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:34.035080 1231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:34.036254 1231 hybrid_clock.cc:648] HybridClock initialized: now 1790506474036168 us; error 97 us; skew 500 ppm
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:34.043622 1231 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:34.044445 1231 webserver.cc:460] Webserver started at http://127.1.46.254:34567/ using document root <none> and password file <none>
I20260927 10:54:34.044950 1231 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:34.045003 1231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:34.045153 1231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:34.047137 1231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "a3041bb7c09e4f689a3efa468e294cb0"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "4d1a9ac0fbecc4598d947cd0492bb8cf"
server_key_iv: "bda40bbe5d59b7c3dbe6443a8deeeac3"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.047616 1231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "a3041bb7c09e4f689a3efa468e294cb0"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "4d1a9ac0fbecc4598d947cd0492bb8cf"
server_key_iv: "bda40bbe5d59b7c3dbe6443a8deeeac3"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.051062 1231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:34.053269 1246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.054476 1231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:34.054610 1231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "a3041bb7c09e4f689a3efa468e294cb0"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "4d1a9ac0fbecc4598d947cd0492bb8cf"
server_key_iv: "bda40bbe5d59b7c3dbe6443a8deeeac3"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.054711 1231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:34.073220 1231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:34.075915 1231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:34.076119 1231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:34.083176 1231 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:36679
I20260927 10:54:34.083205 1298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:36679 every 8 connection(s)
I20260927 10:54:34.084118 1231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:34.086820 1299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:34.091997 1299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0: Bootstrap starting.
I20260927 10:54:34.093654 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 1231
I20260927 10:54:34.093820 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:34.094738 1299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:34.095381 1211 external_mini_cluster.cc:1428] Setting key 6730b0ead1c6ee73a7be56fa630192e5
I20260927 10:54:34.095634 1299 log.cc:824] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:34.097595 1299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0: No bootstrap required, opened a new log
I20260927 10:54:34.101375 1299 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3041bb7c09e4f689a3efa468e294cb0" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36679 } }
I20260927 10:54:34.101613 1299 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:34.101686 1299 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3041bb7c09e4f689a3efa468e294cb0, State: Initialized, Role: FOLLOWER
I20260927 10:54:34.102056 1299 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3041bb7c09e4f689a3efa468e294cb0" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36679 } }
I20260927 10:54:34.102190 1299 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:34.102268 1299 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:34.102392 1299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:34.103440 1299 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3041bb7c09e4f689a3efa468e294cb0" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36679 } }
I20260927 10:54:34.103772 1299 leader_election.cc:302] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3041bb7c09e4f689a3efa468e294cb0; no voters:
I20260927 10:54:34.104090 1299 leader_election.cc:288] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:34.104185 1304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:34.104349 1304 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [term 1 LEADER]: Becoming Leader. State: Replica: a3041bb7c09e4f689a3efa468e294cb0, State: Running, Role: LEADER
I20260927 10:54:34.104627 1304 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3041bb7c09e4f689a3efa468e294cb0" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36679 } }
I20260927 10:54:34.105242 1299 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:34.105676 1304 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3041bb7c09e4f689a3efa468e294cb0. Latest consensus state: current_term: 1 leader_uuid: "a3041bb7c09e4f689a3efa468e294cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3041bb7c09e4f689a3efa468e294cb0" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36679 } } }
I20260927 10:54:34.105775 1304 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:34.106063 1305 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3041bb7c09e4f689a3efa468e294cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3041bb7c09e4f689a3efa468e294cb0" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36679 } } }
I20260927 10:54:34.106163 1305 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:34.106545 1308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:34.109553 1308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:34.115393 1308 catalog_manager.cc:1347] Generated new cluster ID: cbc48890e4b64d0c8e6491b1a9a46ac7
I20260927 10:54:34.115447 1308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:34.121320 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.098453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.1.46.254:36679)
0927 10:54:34.098898 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:34.098917 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:34.100100 (+ 1183us) server_negotiation.cc:407] Connection header received
0927 10:54:34.102854 (+ 2754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:34.102870 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:34.103245 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:34.103587 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:34.106254 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.107935 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.109622 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.110136 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.110308 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.115053 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:34.115062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:34.115076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:34.115105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:34.117271 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.117807 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.117812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.117817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.117929 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.118266 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.118269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.118271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.118732 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:34.118907 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:34.120293 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0927 10:54:34.120561 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":160,"threads_started":1}
I20260927 10:54:34.139313 1308 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:34.140414 1308 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:34.153066 1308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3041bb7c09e4f689a3efa468e294cb0: Generated new TSK 0
I20260927 10:54:34.153729 1308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:34.190150 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36679
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:34.324941 1325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:34.325165 1325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:34.325214 1325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:34.329128 1325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:34.329174 1325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:34.329281 1325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:34.334095 1325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:34.335049 1325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:34.336026 1325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:34.345299 1333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:34.345256 1331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:34.345293 1330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:34.345290 1325 server_base.cc:1034] running on GCE node
I20260927 10:54:34.345851 1325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:34.346439 1325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:34.347622 1325 hybrid_clock.cc:648] HybridClock initialized: now 1790506474347590 us; error 45 us; skew 500 ppm
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:34.350157 1325 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:34.351006 1325 webserver.cc:460] Webserver started at http://127.1.46.193:44679/ using document root <none> and password file <none>
I20260927 10:54:34.351466 1325 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:34.351527 1325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:34.351672 1325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:34.353427 1325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "5eef341980644b22b3e753c2c2457ea3"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "223814dd98f39ca3913a211ca810af2a"
server_key_iv: "658789b8b1575d1ef9cdc84a1cc25e11"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.354023 1325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "5eef341980644b22b3e753c2c2457ea3"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "223814dd98f39ca3913a211ca810af2a"
server_key_iv: "658789b8b1575d1ef9cdc84a1cc25e11"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.359426 1325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 10:54:34.361981 1340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.363179 1325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:34.363335 1325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "5eef341980644b22b3e753c2c2457ea3"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "223814dd98f39ca3913a211ca810af2a"
server_key_iv: "658789b8b1575d1ef9cdc84a1cc25e11"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.363433 1325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:34.373659 1325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:34.376646 1325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:34.376853 1325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:34.377357 1325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:34.378093 1325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:34.378156 1325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.378221 1325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:34.378273 1325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.388226 1325 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:35043
I20260927 10:54:34.388309 1453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:35043 every 8 connection(s)
I20260927 10:54:34.389081 1325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:34.396685 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 1325
I20260927 10:54:34.396811 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:34.397130 1211 external_mini_cluster.cc:1428] Setting key 08123ef7b2d9b689bb100b36823a8500
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:34.403012 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.390876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:40969 (local address 127.1.46.254:36679)
0927 10:54:34.391015 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:34.391018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:34.391618 (+ 600us) server_negotiation.cc:407] Connection header received
0927 10:54:34.393010 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:34.393013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:34.393084 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:34.393185 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:34.394225 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.394850 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.395951 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.396204 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.396316 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.398815 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:34.398819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:34.398822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:34.398848 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:34.400809 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.401421 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.401463 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.401464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.401540 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.401952 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.401954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.401955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.402119 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:34.402191 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:34.402794 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 10:54:34.402895 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:54:34.404232 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.391064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36679 (local address 127.1.46.193:40969)
0927 10:54:34.391473 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:34.391510 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:34.392794 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:34.393346 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:34.393353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:34.393668 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:34.394084 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:34.394097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.394968 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.394974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:34.395826 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:34.395835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.396347 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.396351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:34.396545 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.397091 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:34.397106 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:34.398649 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:34.400931 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:34.400938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:34.400953 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:34.401318 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:34.401647 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:34.401653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:34.401656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:34.401853 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:34.402218 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:34.402227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:34.402510 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 10:54:34.402750 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 10:54:34.403102 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260927 10:54:34.405413 1454 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36679
I20260927 10:54:34.405645 1454 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:34.406250 1454 heartbeater.cc:511] Master 127.1.46.254:36679 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:54:34.407799 1263 ts_manager.cc:194] Registered new tserver with Master: 5eef341980644b22b3e753c2c2457ea3 (127.1.46.193:35043)
I20260927 10:54:34.408943 1263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:40969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:34.424736 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36679
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:34.529072 1459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:34.529359 1459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:34.529446 1459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:34.533316 1459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:34.533398 1459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:34.533507 1459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:34.538383 1459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:34.539317 1459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:34.540222 1459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:34.549219 1467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:34.549204 1464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:34.549204 1465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:34.549434 1459 server_base.cc:1034] running on GCE node
I20260927 10:54:34.550225 1459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:34.550896 1459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:34.552071 1459 hybrid_clock.cc:648] HybridClock initialized: now 1790506474552033 us; error 52 us; skew 500 ppm
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:34.554946 1459 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:34.555817 1459 webserver.cc:460] Webserver started at http://127.1.46.194:39535/ using document root <none> and password file <none>
I20260927 10:54:34.556370 1459 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:34.556460 1459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:34.556699 1459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:34.558851 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "65647c8b01b74f0f9c431266fea40e5d"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "146f201c79dc5bc38fb2a08b9e407eb7"
server_key_iv: "6f3caa9cb9c891e3a71940f0844a38b7"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.559484 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "65647c8b01b74f0f9c431266fea40e5d"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "146f201c79dc5bc38fb2a08b9e407eb7"
server_key_iv: "6f3caa9cb9c891e3a71940f0844a38b7"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.562999 1459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:54:34.565209 1474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.566325 1459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:34.566480 1459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "65647c8b01b74f0f9c431266fea40e5d"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "146f201c79dc5bc38fb2a08b9e407eb7"
server_key_iv: "6f3caa9cb9c891e3a71940f0844a38b7"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.566634 1459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:34.585222 1459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:34.588402 1459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:34.588601 1459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:34.589149 1459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:34.589929 1459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:34.590000 1459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.590071 1459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:34.590114 1459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.599900 1459 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:35351
I20260927 10:54:34.599925 1587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:35351 every 8 connection(s)
I20260927 10:54:34.600733 1459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:34.609861 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 1459
I20260927 10:54:34.609987 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:34.610261 1211 external_mini_cluster.cc:1428] Setting key 3e450a3653f671e9a5988aa1b46a549d
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:34.615492 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.602424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:35985 (local address 127.1.46.254:36679)
0927 10:54:34.602595 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:34.602600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:34.603520 (+ 920us) server_negotiation.cc:407] Connection header received
0927 10:54:34.604820 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:34.604823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:34.604870 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:34.604963 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:34.606268 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.607018 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.608250 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.608472 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.608555 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.611198 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:34.611204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:34.611208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:34.611237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:34.613284 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.613839 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.613845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.613848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.613949 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.614309 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.614315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.614317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.614519 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:34.614674 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:34.615186 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 10:54:34.615311 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:54:34.616227 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.602868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36679 (local address 127.1.46.194:35985)
0927 10:54:34.603362 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:34.603405 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:34.604617 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:34.605090 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:34.605097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:34.605611 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:34.606105 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:34.606120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.607137 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.607140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:34.608105 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:34.608120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.608607 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.608610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:34.608748 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.609271 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:34.609287 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:34.611028 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:34.613396 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:34.613421 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:34.613431 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:34.613730 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:34.614057 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:34.614060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:34.614063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:34.614205 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:34.614670 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:34.614676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:34.614934 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 10:54:34.615127 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 10:54:34.615438 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260927 10:54:34.617414 1588 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36679
I20260927 10:54:34.617681 1588 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:34.618257 1588 heartbeater.cc:511] Master 127.1.46.254:36679 requested a full tablet report, sending...
I20260927 10:54:34.619339 1263 ts_manager.cc:194] Registered new tserver with Master: 65647c8b01b74f0f9c431266fea40e5d (127.1.46.194:35351)
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:54:34.620236 1263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:35985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:34.637066 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36679
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:34.742973 1593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:34.743196 1593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:34.743234 1593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:34.747030 1593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:34.747074 1593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:34.747126 1593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:34.752051 1593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:34.752905 1593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:34.753746 1593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:34.763266 1599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:34.763275 1598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:34.763474 1593 server_base.cc:1034] running on GCE node
W20260927 10:54:34.763291 1601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:34.763932 1593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:34.764485 1593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:34.765672 1593 hybrid_clock.cc:648] HybridClock initialized: now 1790506474765659 us; error 26 us; skew 500 ppm
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:34.768404 1593 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:34.769248 1593 webserver.cc:460] Webserver started at http://127.1.46.195:44979/ using document root <none> and password file <none>
I20260927 10:54:34.769757 1593 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:34.769836 1593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:34.770081 1593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:34.772104 1593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "e7a75dd4b9e54bcbb5b94252a8323bdc"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "56e8cf920fb83864decd24390f76e278"
server_key_iv: "dde7adf7611d7a219a9d4b1c86bb4762"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.772639 1593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "e7a75dd4b9e54bcbb5b94252a8323bdc"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "56e8cf920fb83864decd24390f76e278"
server_key_iv: "dde7adf7611d7a219a9d4b1c86bb4762"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.776053 1593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:54:34.778268 1608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.779517 1593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:34.779639 1593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "e7a75dd4b9e54bcbb5b94252a8323bdc"
format_stamp: "Formatted at 2026-09-27 10:54:34 on dist-test-slave-cffh"
server_key: "56e8cf920fb83864decd24390f76e278"
server_key_iv: "dde7adf7611d7a219a9d4b1c86bb4762"
server_key_version: "encryptionkey@0"
I20260927 10:54:34.779733 1593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:34.798223 1593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:34.801096 1593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:34.801299 1593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:34.801879 1593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:34.802731 1593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:34.802793 1593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.802858 1593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:34.802906 1593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:34.813077 1593 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:43649
I20260927 10:54:34.813103 1721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:43649 every 8 connection(s)
I20260927 10:54:34.813920 1593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:34.822300 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 1593
I20260927 10:54:34.822392 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:34.822737 1211 external_mini_cluster.cc:1428] Setting key 7cc2e5b82592124ef4e70e13255cc852
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:34.827135 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.815625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:39497 (local address 127.1.46.254:36679)
0927 10:54:34.815753 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:34.815756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:34.816465 (+ 709us) server_negotiation.cc:407] Connection header received
0927 10:54:34.817777 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:34.817781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:34.817840 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:34.817921 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:34.819218 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.819761 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.820892 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.821129 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.821240 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.823825 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:34.823829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:34.823831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:34.823859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:34.825385 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.825849 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.825851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.825852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.825919 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.826153 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.826155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.826156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.826341 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:34.826429 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:34.826933 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 10:54:34.827028 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:54:34.827950 1724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.815922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36679 (local address 127.1.46.195:39497)
0927 10:54:34.816324 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:34.816363 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:34.817571 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:34.818092 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:34.818098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:34.818605 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:34.819072 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:34.819084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.819881 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.819887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:34.820756 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:34.820765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.821272 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.821277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:34.821443 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.822026 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:34.822042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:34.823658 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:34.825504 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:34.825509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:34.825519 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:34.825767 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:34.825976 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:34.825977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:34.825978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:34.826084 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:34.826440 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:34.826445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:34.826731 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 10:54:34.826883 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 10:54:34.827106 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":116,"threads_started":1}
I20260927 10:54:34.828966 1722 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36679
I20260927 10:54:34.829197 1722 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:34.829648 1722 heartbeater.cc:511] Master 127.1.46.254:36679 requested a full tablet report, sending...
I20260927 10:54:34.830639 1263 ts_manager.cc:194] Registered new tserver with Master: e7a75dd4b9e54bcbb5b94252a8323bdc (127.1.46.195:43649)
I20260927 10:54:34.831177 1263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:39497
I20260927 10:54:34.836871 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:54:34.862025 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.852619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.1.46.193:35043)
0927 10:54:34.852953 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:34.852967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:34.852993 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:54:34.853180 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:34.853187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:34.853372 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:34.853527 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:34.854194 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.855037 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.855867 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.856231 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.856345 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.858881 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:34.858890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:34.858903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:34.858951 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:34.860412 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.860827 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.860832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.860833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.860910 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.861145 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.861147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.861148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.861369 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:34.861575 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:34.861748 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 10:54:34.861870 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":130,"threads_started":1}
W20260927 10:54:34.863031 1368 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47274
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:54:34.889142 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:34.880197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.1.46.193:35043)
0927 10:54:34.880332 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:34.880337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:34.880521 (+ 184us) server_negotiation.cc:407] Connection header received
0927 10:54:34.880668 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:34.880676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:34.880720 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:34.880807 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:34.881487 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.881992 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.882868 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:34.883155 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:34.883301 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:34.885714 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:34.885717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:34.885719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:34.885742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:34.887481 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.887920 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.887925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.887928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.888046 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:34.888323 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:34.888327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:34.888330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:34.888494 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:34.888622 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:34.888852 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 10:54:34.888961 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:54:34.890193 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 1325
I20260927 10:54:34.897115 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 1459
I20260927 10:54:34.903223 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 1593
I20260927 10:54:34.908329 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 1231
2026-09-27T10:54:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2152 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1736](info): set up 2 sockets
Sep 27 10:54:34 dist-test-slave-cffh krb5kdc[1736](info): commencing operation
krb5kdc: starting...
W20260927 10:54:35.954238 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:36.069176 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44017
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:44017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:36.173635 1749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:36.173879 1749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:36.173918 1749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:36.177700 1749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:36.177745 1749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:36.177763 1749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:36.177779 1749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:36.177793 1749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:36.182493 1749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:44017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:36.183410 1749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:36.184332 1749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:36.192207 1755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.192298 1754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.192256 1757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:36.192394 1749 server_base.cc:1034] running on GCE node
I20260927 10:54:36.193051 1749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:36.194190 1749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:36.195374 1749 hybrid_clock.cc:648] HybridClock initialized: now 1790506476195350 us; error 35 us; skew 500 ppm
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:36.197926 1749 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:36.198836 1749 webserver.cc:460] Webserver started at http://127.1.46.254:36493/ using document root <none> and password file <none>
I20260927 10:54:36.199349 1749 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:36.199433 1749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:36.199669 1749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:36.201550 1749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "98d67f0210b2472bbcc5f489962384dd"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "4b6aa36e957a9c99410379cbd82ca23e"
server_key_iv: "012cab3fd14886d4c172af38373b2d42"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.202076 1749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "98d67f0210b2472bbcc5f489962384dd"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "4b6aa36e957a9c99410379cbd82ca23e"
server_key_iv: "012cab3fd14886d4c172af38373b2d42"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.205572 1749 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:54:36.207783 1764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.208839 1749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:36.208967 1749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "98d67f0210b2472bbcc5f489962384dd"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "4b6aa36e957a9c99410379cbd82ca23e"
server_key_iv: "012cab3fd14886d4c172af38373b2d42"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.209074 1749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:36.222266 1749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:36.224982 1749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:36.225178 1749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:36.231873 1749 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:44017
I20260927 10:54:36.231884 1816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:44017 every 8 connection(s)
I20260927 10:54:36.232734 1749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:36.234064 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 1749
I20260927 10:54:36.234189 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:36.234431 1211 external_mini_cluster.cc:1428] Setting key 61408944bf50b6b36b2953e1f2068814
I20260927 10:54:36.235698 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:36.241825 1817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd: Bootstrap starting.
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:36.244330 1817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:36.245194 1817 log.cc:824] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:36.247339 1817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd: No bootstrap required, opened a new log
I20260927 10:54:36.248312 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.235703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.1.46.254:44017)
0927 10:54:36.236119 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.236143 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.236176 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:54:36.236785 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.236800 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.237100 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.237352 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:36.238205 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.239253 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.240356 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.240778 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.240888 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.243516 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:36.243521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:36.243536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:36.243574 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:36.245400 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.245925 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.245930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.245935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.246057 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.246328 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.246331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.246333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.246812 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:36.246988 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:36.247183 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:54:36.247457 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":130,"threads_started":1}
I20260927 10:54:36.251236 1817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d67f0210b2472bbcc5f489962384dd" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44017 } }
I20260927 10:54:36.251438 1817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:36.251475 1817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d67f0210b2472bbcc5f489962384dd, State: Initialized, Role: FOLLOWER
I20260927 10:54:36.251847 1817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d67f0210b2472bbcc5f489962384dd" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44017 } }
I20260927 10:54:36.251956 1817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:36.252002 1817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:36.252075 1817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:36.252780 1817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d67f0210b2472bbcc5f489962384dd" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44017 } }
I20260927 10:54:36.253100 1817 leader_election.cc:302] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98d67f0210b2472bbcc5f489962384dd; no voters:
I20260927 10:54:36.253412 1817 leader_election.cc:288] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:36.253512 1822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:36.253736 1822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [term 1 LEADER]: Becoming Leader. State: Replica: 98d67f0210b2472bbcc5f489962384dd, State: Running, Role: LEADER
I20260927 10:54:36.254046 1822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d67f0210b2472bbcc5f489962384dd" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44017 } }
I20260927 10:54:36.254511 1817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:36.255767 1824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98d67f0210b2472bbcc5f489962384dd. Latest consensus state: current_term: 1 leader_uuid: "98d67f0210b2472bbcc5f489962384dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d67f0210b2472bbcc5f489962384dd" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44017 } } }
I20260927 10:54:36.255940 1824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:36.255767 1823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98d67f0210b2472bbcc5f489962384dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d67f0210b2472bbcc5f489962384dd" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44017 } } }
I20260927 10:54:36.256070 1823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:36.256371 1834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:36.259303 1834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:36.264545 1834 catalog_manager.cc:1347] Generated new cluster ID: 34483db81f3a43a9ba28d6641c6213fd
I20260927 10:54:36.264604 1834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:36.277144 1834 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:36.278052 1834 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:36.286981 1834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98d67f0210b2472bbcc5f489962384dd: Generated new TSK 0
I20260927 10:54:36.287640 1834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:36.316972 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44017
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:36.439498 1843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:36.439724 1843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:36.439764 1843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:36.443555 1843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:36.443606 1843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:36.443689 1843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:36.448565 1843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:36.449496 1843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:36.450487 1843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:36.459374 1851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.459385 1849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.459369 1848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:36.459630 1843 server_base.cc:1034] running on GCE node
I20260927 10:54:36.460053 1843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:36.460690 1843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:36.461887 1843 hybrid_clock.cc:648] HybridClock initialized: now 1790506476461868 us; error 34 us; skew 500 ppm
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:36.464681 1843 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:36.465464 1843 webserver.cc:460] Webserver started at http://127.1.46.193:39203/ using document root <none> and password file <none>
I20260927 10:54:36.465975 1843 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:36.466049 1843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:36.466210 1843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:36.468358 1843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "e7a9595dff9c43f0aa501bc97ed5da81"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "a30f27a96cb3d15ee5fb1497b450a845"
server_key_iv: "61752f7dbce59f4ec9c5815f94f88855"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.468875 1843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "e7a9595dff9c43f0aa501bc97ed5da81"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "a30f27a96cb3d15ee5fb1497b450a845"
server_key_iv: "61752f7dbce59f4ec9c5815f94f88855"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.472549 1843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:54:36.474969 1858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.476156 1843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:36.476250 1843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "e7a9595dff9c43f0aa501bc97ed5da81"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "a30f27a96cb3d15ee5fb1497b450a845"
server_key_iv: "61752f7dbce59f4ec9c5815f94f88855"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.476377 1843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:36.516552 1843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:36.519542 1843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:36.519696 1843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:36.520229 1843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:36.521052 1843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:36.521098 1843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.521138 1843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:36.521162 1843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.530128 1843 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:46109
I20260927 10:54:36.530201 1971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:46109 every 8 connection(s)
I20260927 10:54:36.531072 1843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:36.537838 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 1843
I20260927 10:54:36.537946 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:36.538225 1211 external_mini_cluster.cc:1428] Setting key 89250d834699fb74cfd13ebd9e7a826f
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:36.545660 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.532715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:59667 (local address 127.1.46.254:44017)
0927 10:54:36.532838 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.532842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.533703 (+ 861us) server_negotiation.cc:407] Connection header received
0927 10:54:36.535083 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.535086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.535145 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.535252 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:36.536316 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.536847 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.538151 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.538453 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.538643 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.541559 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:36.541574 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:36.541578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:36.541611 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:36.543419 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.543919 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.543926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.543930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.544025 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.544425 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.544431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.544435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.544682 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:36.544793 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:36.545273 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 10:54:36.545415 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:54:36.546216 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.533141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44017 (local address 127.1.46.193:59667)
0927 10:54:36.533560 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:36.533602 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:36.534883 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:36.535411 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:36.535417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:36.535740 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:36.536174 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.536186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.536966 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.536973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.538013 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.538026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.538663 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.538668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.538855 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.539408 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:36.539423 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:36.541368 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:36.543544 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:36.543548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:36.543559 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:36.543815 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:36.544136 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:36.544143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:36.544147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:36.544285 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:36.544813 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:36.544818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:36.545036 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 10:54:36.545190 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 10:54:36.545422 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260927 10:54:36.547356 1972 heartbeater.cc:347] Connected to a master server at 127.1.46.254:44017
I20260927 10:54:36.547596 1972 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:36.548168 1972 heartbeater.cc:511] Master 127.1.46.254:44017 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:54:36.549645 1781 ts_manager.cc:194] Registered new tserver with Master: e7a9595dff9c43f0aa501bc97ed5da81 (127.1.46.193:46109)
I20260927 10:54:36.550855 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:59667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:36.565068 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44017
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:36.670187 1977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:36.670433 1977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:36.670523 1977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:36.674324 1977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:36.674368 1977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:36.674459 1977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:36.679395 1977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:36.680361 1977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:36.681335 1977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:36.689874 1983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.689963 1982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.689935 1985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:36.690142 1977 server_base.cc:1034] running on GCE node
I20260927 10:54:36.690560 1977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:36.691171 1977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:36.692353 1977 hybrid_clock.cc:648] HybridClock initialized: now 1790506476692342 us; error 27 us; skew 500 ppm
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:36.694973 1977 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:36.695808 1977 webserver.cc:460] Webserver started at http://127.1.46.194:45347/ using document root <none> and password file <none>
I20260927 10:54:36.696305 1977 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:36.696388 1977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:36.696589 1977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:36.698345 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "06d1df78c2f14e4e93da2adff52275e9"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "75a51e716a8d8d86dea1424034a997ee"
server_key_iv: "eb7fc5718ac68fa361cfb9a9f56f1b79"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.698892 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "06d1df78c2f14e4e93da2adff52275e9"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "75a51e716a8d8d86dea1424034a997ee"
server_key_iv: "eb7fc5718ac68fa361cfb9a9f56f1b79"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.702236 1977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 10:54:36.704613 1992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.705706 1977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:36.705826 1977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "06d1df78c2f14e4e93da2adff52275e9"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "75a51e716a8d8d86dea1424034a997ee"
server_key_iv: "eb7fc5718ac68fa361cfb9a9f56f1b79"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.705924 1977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:36.719863 1977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:36.722525 1977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:36.722715 1977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:36.723191 1977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:36.723918 1977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:36.723984 1977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.724046 1977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:36.724094 1977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.732988 1977 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:42389
I20260927 10:54:36.733024 2105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:42389 every 8 connection(s)
I20260927 10:54:36.733747 1977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:36.740115 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 1977
I20260927 10:54:36.740208 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:36.740446 1211 external_mini_cluster.cc:1428] Setting key 5f8f345b40a7a7acf48b686a1e83bdc4
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:36.747771 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.735413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:33615 (local address 127.1.46.254:44017)
0927 10:54:36.735550 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.735555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.736349 (+ 794us) server_negotiation.cc:407] Connection header received
0927 10:54:36.737575 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.737577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.737620 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.737691 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:36.738866 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.739393 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.740719 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.741018 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.741134 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.743820 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:36.743823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:36.743825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:36.743845 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:36.745279 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.745936 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.745938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.745939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.746041 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.746466 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.746468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.746470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.746680 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:36.746755 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:36.747454 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 10:54:36.747580 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:54:36.748893 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.735763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44017 (local address 127.1.46.194:33615)
0927 10:54:36.736195 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:36.736232 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:36.737377 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:36.737838 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:36.737844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:36.738204 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:36.738713 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.738727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.739517 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.739520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.740596 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.740606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.741185 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.741189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.741370 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.742015 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:36.742031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:36.743657 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:36.745394 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:36.745403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:36.745421 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:36.745832 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:36.746161 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:36.746166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:36.746171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:36.746360 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:36.746787 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:36.746798 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:36.747140 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 10:54:36.747410 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 10:54:36.747802 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
I20260927 10:54:36.749946 2106 heartbeater.cc:347] Connected to a master server at 127.1.46.254:44017
I20260927 10:54:36.750169 2106 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:54:36.750660 2106 heartbeater.cc:511] Master 127.1.46.254:44017 requested a full tablet report, sending...
I20260927 10:54:36.751641 1781 ts_manager.cc:194] Registered new tserver with Master: 06d1df78c2f14e4e93da2adff52275e9 (127.1.46.194:42389)
I20260927 10:54:36.752135 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:33615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:36.768913 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44017
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:36.880471 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:36.880759 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:36.880831 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:36.884815 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:36.884894 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:36.884980 2111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:36.890069 2111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:36.891126 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:36.892092 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:36.901129 2119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.901216 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:36.901170 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:36.901288 2111 server_base.cc:1034] running on GCE node
I20260927 10:54:36.901747 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:36.902381 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:36.903575 2111 hybrid_clock.cc:648] HybridClock initialized: now 1790506476903544 us; error 48 us; skew 500 ppm
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:36.906086 2111 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:36.906946 2111 webserver.cc:460] Webserver started at http://127.1.46.195:37703/ using document root <none> and password file <none>
I20260927 10:54:36.907410 2111 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:36.907464 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:36.907617 2111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:36.909406 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "43c56a38144c4587ae1e5552305e5686"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "dccbdb1171eedbd12aedc151ba1364c4"
server_key_iv: "4c212d7fed42c01d664f14e4015d3973"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.909962 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "43c56a38144c4587ae1e5552305e5686"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "dccbdb1171eedbd12aedc151ba1364c4"
server_key_iv: "4c212d7fed42c01d664f14e4015d3973"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.913785 2111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:54:36.916038 2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.917232 2111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:36.917363 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "43c56a38144c4587ae1e5552305e5686"
format_stamp: "Formatted at 2026-09-27 10:54:36 on dist-test-slave-cffh"
server_key: "dccbdb1171eedbd12aedc151ba1364c4"
server_key_iv: "4c212d7fed42c01d664f14e4015d3973"
server_key_version: "encryptionkey@0"
I20260927 10:54:36.917466 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:36.934124 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:36.936821 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:36.937001 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:36.937551 2111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:36.938314 2111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:36.938356 2111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.938438 2111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:36.938472 2111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:36.947451 2111 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:36035
I20260927 10:54:36.947472 2239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:36035 every 8 connection(s)
I20260927 10:54:36.948276 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:36.954241 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 2111
I20260927 10:54:36.954355 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:36.954679 1211 external_mini_cluster.cc:1428] Setting key f6e1f13b5bc4f1fb00c7eb7b90394eee
Sep 27 10:54:36 dist-test-slave-cffh krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:36.961514 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.949968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:52149 (local address 127.1.46.254:44017)
0927 10:54:36.950126 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.950131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.950919 (+ 788us) server_negotiation.cc:407] Connection header received
0927 10:54:36.952139 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.952141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.952186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.952258 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:36.953282 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.953784 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.955336 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.955815 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.955946 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.958521 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:36.958524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:36.958526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:36.958547 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:36.959803 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.960272 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.960275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.960276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.960340 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.960646 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.960648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.960650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.960800 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:36.960888 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:36.961321 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 10:54:36.961406 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:54:36.962337 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.950346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44017 (local address 127.1.46.195:52149)
0927 10:54:36.950788 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:36.950826 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:36.951950 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:36.952404 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:36.952410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:36.952723 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:36.953163 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.953175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.953879 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.953881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.955163 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.955195 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.955985 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.955989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.956194 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.956812 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:36.956832 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:36.958366 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:36.959920 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:36.959925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:36.959935 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:36.960183 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:36.960416 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:36.960418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:36.960420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:36.960553 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:36.960901 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:36.960906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:36.961127 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 10:54:36.961283 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 10:54:36.961537 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":102,"threads_started":1}
I20260927 10:54:36.963383 2240 heartbeater.cc:347] Connected to a master server at 127.1.46.254:44017
I20260927 10:54:36.963651 2240 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:36.964100 2240 heartbeater.cc:511] Master 127.1.46.254:44017 requested a full tablet report, sending...
I20260927 10:54:36.965015 1781 ts_manager.cc:194] Registered new tserver with Master: 43c56a38144c4587ae1e5552305e5686 (127.1.46.195:36035)
I20260927 10:54:36.965544 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:52149
I20260927 10:54:36.968457 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:54:36.978642 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.971862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.1.46.254:44017)
0927 10:54:36.971973 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.971976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.972192 (+ 216us) server_negotiation.cc:407] Connection header received
0927 10:54:36.972284 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.972286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.972319 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.972376 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:36.973011 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.973468 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.974293 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.974515 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.974668 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.975863 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:36.975867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:36.975871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:36.975894 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:36.977010 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.977414 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.977418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.977421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.977495 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:36.977767 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:36.977771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:36.977773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:36.977962 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:36.978096 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:36.978334 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 10:54:36.978442 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:54:36.983191 1781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:36.986303 1781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:37.000037 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.994348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:36035 (local address 127.0.0.1:58240)
0927 10:54:36.995293 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:36.995307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:36.995394 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:36.996117 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:36.996124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:36.996149 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:36.996277 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.996284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.997351 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.997356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.998442 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.998453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.999513 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.999516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.999622 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.999661 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 10:54:36.999776 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:54:36.999898 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":6,"spinlock_wait_cycles":6016,"thread_start_us":86,"threads_started":1}
I20260927 10:54:37.000072 2254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.994628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.1.46.195:36035)
0927 10:54:36.995631 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.995642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.995663 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:54:36.995740 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.995746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.995951 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.996105 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:36.996410 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.997220 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.998604 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.999394 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.999500 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:36.999662 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:36.999841 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 10:54:36.999929 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":125,"threads_started":1}
I20260927 10:54:37.000458 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.994621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:46109 (local address 127.0.0.1:55690)
0927 10:54:36.995554 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:36.995568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:36.995651 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:36.996329 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:36.996335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:36.996354 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:36.996512 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.996518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.998034 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.998037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.999002 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.999011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.000080 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.000084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:37.000192 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.000223 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:54:37.000294 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:54:37.000368 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":63,"spinlock_wait_cycles":3328,"thread_start_us":81,"threads_started":1}
I20260927 10:54:37.000882 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.995108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.1.46.193:46109)
0927 10:54:36.995487 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.995503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.995742 (+ 239us) server_negotiation.cc:407] Connection header received
0927 10:54:36.995821 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.995826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.996018 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.996183 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:36.996642 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.997819 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.999128 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.999811 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.999904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.000020 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:37.000697 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 10:54:37.000779 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":91,"threads_started":1}
I20260927 10:54:37.001130 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.994339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:42389 (local address 127.0.0.1:36686)
0927 10:54:36.995131 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:36.995182 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:36.995300 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:36.996367 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:36.996370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:36.996397 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:36.996549 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.996555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.997955 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.997958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:36.999518 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:36.999528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.000764 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.000767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:37.000846 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.000865 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:54:37.000940 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:54:37.000997 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":385,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260927 10:54:37.001991 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:36.995014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.1.46.194:42389)
0927 10:54:36.995683 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:36.995695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:36.995712 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:54:36.995771 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:36.995775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:36.996215 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:36.996386 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:36.996694 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:36.997819 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:36.999645 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.000487 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.000587 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.000707 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:37.001784 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0927 10:54:37.001867 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":104,"threads_started":1}
I20260927 10:54:37.003223 2174 tablet_service.cc:1467] Processing CreateTablet for tablet d34da5ab6d6a462fb5f5b695d9f72493 (DEFAULT_TABLE table=test-table [id=80b55e7496bd4a4780863a39484cc724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:37.003518 1906 tablet_service.cc:1467] Processing CreateTablet for tablet d34da5ab6d6a462fb5f5b695d9f72493 (DEFAULT_TABLE table=test-table [id=80b55e7496bd4a4780863a39484cc724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:37.004244 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34da5ab6d6a462fb5f5b695d9f72493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:37.004441 1906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34da5ab6d6a462fb5f5b695d9f72493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:37.008571 2040 tablet_service.cc:1467] Processing CreateTablet for tablet d34da5ab6d6a462fb5f5b695d9f72493 (DEFAULT_TABLE table=test-table [id=80b55e7496bd4a4780863a39484cc724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:37.009507 2040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34da5ab6d6a462fb5f5b695d9f72493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:37.010234 2257 tablet_bootstrap.cc:492] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81: Bootstrap starting.
I20260927 10:54:37.011755 2258 tablet_bootstrap.cc:492] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686: Bootstrap starting.
I20260927 10:54:37.012753 2257 tablet_bootstrap.cc:654] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:37.013994 2258 tablet_bootstrap.cc:654] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:37.014008 2257 log.cc:824] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:37.015225 2258 log.cc:824] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:37.015386 2260 tablet_bootstrap.cc:492] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9: Bootstrap starting.
I20260927 10:54:37.016327 2257 tablet_bootstrap.cc:492] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81: No bootstrap required, opened a new log
I20260927 10:54:37.016500 2257 ts_tablet_manager.cc:1397] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:54:37.017195 2258 tablet_bootstrap.cc:492] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686: No bootstrap required, opened a new log
I20260927 10:54:37.017412 2258 ts_tablet_manager.cc:1397] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:54:37.017779 2260 tablet_bootstrap.cc:654] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:37.018872 2260 log.cc:824] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:37.020413 2260 tablet_bootstrap.cc:492] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9: No bootstrap required, opened a new log
I20260927 10:54:37.020677 2260 ts_tablet_manager.cc:1397] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 10:54:37.020498 2257 raft_consensus.cc:348] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.020749 2257 raft_consensus.cc:374] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:37.020857 2257 raft_consensus.cc:729] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a9595dff9c43f0aa501bc97ed5da81, State: Initialized, Role: FOLLOWER
I20260927 10:54:37.021075 2258 raft_consensus.cc:348] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.021337 2258 raft_consensus.cc:374] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:37.021306 2257 consensus_queue.cc:260] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.021432 2258 raft_consensus.cc:729] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c56a38144c4587ae1e5552305e5686, State: Initialized, Role: FOLLOWER
I20260927 10:54:37.021903 2258 consensus_queue.cc:260] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.022006 1972 heartbeater.cc:503] Master 127.1.46.254:44017 was elected leader, sending a full tablet report...
I20260927 10:54:37.022316 2257 ts_tablet_manager.cc:1428] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:54:37.022588 2240 heartbeater.cc:503] Master 127.1.46.254:44017 was elected leader, sending a full tablet report...
I20260927 10:54:37.023794 2258 ts_tablet_manager.cc:1428] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:54:37.024951 2260 raft_consensus.cc:348] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.025207 2260 raft_consensus.cc:374] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:37.025303 2260 raft_consensus.cc:729] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d1df78c2f14e4e93da2adff52275e9, State: Initialized, Role: FOLLOWER
I20260927 10:54:37.025679 2260 consensus_queue.cc:260] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.026319 2106 heartbeater.cc:503] Master 127.1.46.254:44017 was elected leader, sending a full tablet report...
I20260927 10:54:37.026552 2260 ts_tablet_manager.cc:1428] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
W20260927 10:54:37.032171 1973 tablet.cc:2369] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:54:37.199211 2241 tablet.cc:2369] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:54:37.214442 2264 raft_consensus.cc:482] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:37.214681 2264 raft_consensus.cc:504] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.215905 2264 leader_election.cc:288] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d1df78c2f14e4e93da2adff52275e9 (127.1.46.194:42389), e7a9595dff9c43f0aa501bc97ed5da81 (127.1.46.193:46109)
I20260927 10:54:37.219988 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:37.216226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:46211 (local address 127.1.46.193:46109)
0927 10:54:37.216339 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:37.216344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:37.216369 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:54:37.216419 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:37.216421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:37.216463 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:37.216529 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:37.216857 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.217559 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.218936 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.219488 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.219579 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.219625 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:37.219844 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 10:54:37.219892 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 10:54:37.220072 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:37.216136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:46109 (local address 127.1.46.195:46211)
0927 10:54:37.216246 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:37.216262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:37.216354 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:37.216599 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:37.216603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:37.216623 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:37.216739 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:37.216744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.217681 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.217685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:37.218822 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:37.218832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.219599 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.219603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:37.219720 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.219747 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:54:37.219833 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:54:37.219907 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 10:54:37.220721 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:37.216139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:42389 (local address 127.1.46.195:42575)
0927 10:54:37.216770 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:37.216781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:37.216898 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:37.217142 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:37.217145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:37.217162 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:37.217280 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:37.217286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.218272 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.218275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:37.219263 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:37.219272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.220266 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.220269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:37.220390 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.220414 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:54:37.220504 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:54:37.220574 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":268,"thread_start_us":158,"threads_started":1}
I20260927 10:54:37.220781 1926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34da5ab6d6a462fb5f5b695d9f72493" candidate_uuid: "43c56a38144c4587ae1e5552305e5686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" is_pre_election: true
I20260927 10:54:37.221117 1926 raft_consensus.cc:2454] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c56a38144c4587ae1e5552305e5686 in term 0.
I20260927 10:54:37.221174 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:37.216198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:42575 (local address 127.1.46.194:42389)
0927 10:54:37.216310 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:37.216314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:37.216908 (+ 594us) server_negotiation.cc:407] Connection header received
0927 10:54:37.216969 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:37.216971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:37.217020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:37.217096 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:37.217391 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.218153 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.219386 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.220007 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.220132 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.220188 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:37.221001 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 10:54:37.221082 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:54:37.221817 2060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34da5ab6d6a462fb5f5b695d9f72493" candidate_uuid: "43c56a38144c4587ae1e5552305e5686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d1df78c2f14e4e93da2adff52275e9" is_pre_election: true
I20260927 10:54:37.221871 2127 leader_election.cc:302] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c56a38144c4587ae1e5552305e5686, e7a9595dff9c43f0aa501bc97ed5da81; no voters:
I20260927 10:54:37.222164 2060 raft_consensus.cc:2454] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c56a38144c4587ae1e5552305e5686 in term 0.
I20260927 10:54:37.222240 2264 raft_consensus.cc:2781] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:37.222355 2264 raft_consensus.cc:482] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:37.222482 2264 raft_consensus.cc:3037] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:37.224246 2264 raft_consensus.cc:504] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.224643 2264 leader_election.cc:288] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [CANDIDATE]: Term 1 election: Requested vote from peers 06d1df78c2f14e4e93da2adff52275e9 (127.1.46.194:42389), e7a9595dff9c43f0aa501bc97ed5da81 (127.1.46.193:46109)
I20260927 10:54:37.224941 2060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34da5ab6d6a462fb5f5b695d9f72493" candidate_uuid: "43c56a38144c4587ae1e5552305e5686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d1df78c2f14e4e93da2adff52275e9"
I20260927 10:54:37.224972 1926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34da5ab6d6a462fb5f5b695d9f72493" candidate_uuid: "43c56a38144c4587ae1e5552305e5686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a9595dff9c43f0aa501bc97ed5da81"
I20260927 10:54:37.225088 1926 raft_consensus.cc:3037] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:37.225106 2060 raft_consensus.cc:3037] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:37.226118 1926 raft_consensus.cc:2454] T d34da5ab6d6a462fb5f5b695d9f72493 P e7a9595dff9c43f0aa501bc97ed5da81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c56a38144c4587ae1e5552305e5686 in term 1.
I20260927 10:54:37.226224 2060 raft_consensus.cc:2454] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c56a38144c4587ae1e5552305e5686 in term 1.
I20260927 10:54:37.226393 2127 leader_election.cc:302] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c56a38144c4587ae1e5552305e5686, e7a9595dff9c43f0aa501bc97ed5da81; no voters:
I20260927 10:54:37.226634 2264 raft_consensus.cc:2781] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:37.226892 2264 raft_consensus.cc:686] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [term 1 LEADER]: Becoming Leader. State: Replica: 43c56a38144c4587ae1e5552305e5686, State: Running, Role: LEADER
I20260927 10:54:37.227221 2264 consensus_queue.cc:237] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } }
I20260927 10:54:37.230253 1781 catalog_manager.cc:5495] T d34da5ab6d6a462fb5f5b695d9f72493 P 43c56a38144c4587ae1e5552305e5686 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43c56a38144c4587ae1e5552305e5686 (127.1.46.195). New cstate: current_term: 1 leader_uuid: "43c56a38144c4587ae1e5552305e5686" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d1df78c2f14e4e93da2adff52275e9" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c56a38144c4587ae1e5552305e5686" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 36035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7a9595dff9c43f0aa501bc97ed5da81" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 46109 } health_report { overall_health: UNKNOWN } } }
W20260927 10:54:37.234685 2107 tablet.cc:2369] T d34da5ab6d6a462fb5f5b695d9f72493 P 06d1df78c2f14e4e93da2adff52275e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:54:37.292784 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:37.283672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.1.46.193:46109)
0927 10:54:37.283843 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:37.283848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:37.284052 (+ 204us) server_negotiation.cc:407] Connection header received
0927 10:54:37.284138 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:37.284140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:37.284185 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:37.284260 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:37.284887 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.285409 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.286173 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.286424 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.286523 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.289127 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:37.289140 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:37.289156 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:37.289179 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:37.291073 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:37.291478 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:37.291484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:37.291485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:37.291606 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:37.291865 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:37.291869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:37.291870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:37.292129 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:37.292297 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:37.292480 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 10:54:37.292639 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506477, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506477, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:54:37.319188 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:37.310709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.1.46.193:46109)
0927 10:54:37.310925 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:37.310930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:37.311069 (+ 139us) server_negotiation.cc:407] Connection header received
0927 10:54:37.311228 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:37.311233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:37.311283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:37.311372 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:37.312071 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.312660 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.313549 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.313791 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.313900 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.316268 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:37.316271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:37.316273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:37.316294 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:37.317509 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:37.317927 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:37.317932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:37.317935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:37.318012 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:37.318286 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:37.318290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:37.318292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:37.318448 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:37.318621 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:37.318907 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 10:54:37.319025 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:54:37.344213 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:37.336139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.1.46.193:46109)
0927 10:54:37.336283 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:37.336287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:37.336451 (+ 164us) server_negotiation.cc:407] Connection header received
0927 10:54:37.336594 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:37.336598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:37.336648 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:37.336733 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:37.337356 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.337838 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.338712 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:37.338941 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:37.339048 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:37.341338 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:37.341349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:37.341352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:37.341372 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:37.342624 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:37.343040 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:37.343044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:37.343047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:37.343129 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:37.343403 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:37.343409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:37.343411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:37.343563 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:37.343710 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:37.343939 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 10:54:37.344049 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 10:54:37.345563 1886 tablet_service.cc:2908] Rejecting scan request for tablet d34da5ab6d6a462fb5f5b695d9f72493: Uninitialized: clean time has not yet been initialized
I20260927 10:54:37.348120 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 1843
I20260927 10:54:37.354115 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 1977
I20260927 10:54:37.359555 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 2111
I20260927 10:54:37.364748 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 1749
2026-09-27T10:54:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2456 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[2281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[2281](info): set up 2 sockets
Sep 27 10:54:37 dist-test-slave-cffh krb5kdc[2281](info): commencing operation
krb5kdc: starting...
W20260927 10:54:38.394415 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:38 dist-test-slave-cffh krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:38.508940 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:37467
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:38.611562 2294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:38.611843 2294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:38.611915 2294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:38.615873 2294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:38.615947 2294 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:54:38.615978 2294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:38.616039 2294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:38.616078 2294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:38.616114 2294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:38.621325 2294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:37467
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:38.622327 2294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:38.623344 2294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:38.636801 2302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:38.636801 2300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:38.636844 2299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:38.638442 2294 server_base.cc:1034] running on GCE node
I20260927 10:54:38.639227 2294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:38.640590 2294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:38.641777 2294 hybrid_clock.cc:648] HybridClock initialized: now 1790506478641748 us; error 41 us; skew 500 ppm
Sep 27 10:54:38 dist-test-slave-cffh krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:38.644958 2294 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:38.646170 2294 webserver.cc:460] Webserver started at http://127.1.46.254:45457/ using document root <none> and password file <none>
I20260927 10:54:38.646924 2294 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:38.647010 2294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:38.647233 2294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:38.649770 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8"
format_stamp: "Formatted at 2026-09-27 10:54:38 on dist-test-slave-cffh"
server_key: "234fcf692d4a6497044e74c8804f0bad"
server_key_iv: "5b80e65d71dde79301f7220bd45f337d"
server_key_version: "encryptionkey@0"
I20260927 10:54:38.650444 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8"
format_stamp: "Formatted at 2026-09-27 10:54:38 on dist-test-slave-cffh"
server_key: "234fcf692d4a6497044e74c8804f0bad"
server_key_iv: "5b80e65d71dde79301f7220bd45f337d"
server_key_version: "encryptionkey@0"
I20260927 10:54:38.655586 2294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 10:54:38.658514 2309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:38.659884 2294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:38.660010 2294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8"
format_stamp: "Formatted at 2026-09-27 10:54:38 on dist-test-slave-cffh"
server_key: "234fcf692d4a6497044e74c8804f0bad"
server_key_iv: "5b80e65d71dde79301f7220bd45f337d"
server_key_version: "encryptionkey@0"
I20260927 10:54:38.660156 2294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:38.688463 2294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:38.691560 2294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:38.691766 2294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:38.699509 2294 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:37467
I20260927 10:54:38.699530 2371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:37467 every 8 connection(s)
I20260927 10:54:38.700379 2294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:38.703162 2372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:38.704246 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 2294
I20260927 10:54:38.704380 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:38.704687 1211 external_mini_cluster.cc:1428] Setting key 0965e54307604ebd2e645ee2aa652187
I20260927 10:54:38.709175 2372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8: Bootstrap starting.
I20260927 10:54:38.711376 2372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:38.712132 2372 log.cc:824] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:54:38 dist-test-slave-cffh krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:38.714169 2372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8: No bootstrap required, opened a new log
I20260927 10:54:38.717198 2372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37467 } }
I20260927 10:54:38.717389 2372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:38.717470 2372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8ba525583b4d5e9fe97dfcbd50ecf8, State: Initialized, Role: FOLLOWER
I20260927 10:54:38.717923 2372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37467 } }
I20260927 10:54:38.718039 2372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:38.718101 2372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:38.718209 2372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:38.718990 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:38.706120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.1.46.254:37467)
0927 10:54:38.706658 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:38.706673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:38.706705 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:54:38.707324 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:38.707339 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:38.707661 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:38.707983 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:38.708752 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.709690 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.710524 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.711017 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.711200 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:38.713820 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:38.713825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:38.713840 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:38.713869 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:38.715775 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:38.716278 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:38.716285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:38.716298 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:38.716409 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:38.716731 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:38.716736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:38.716738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:38.717092 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:38.717292 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:38.717885 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 10:54:38.718144 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":157,"threads_started":1}
I20260927 10:54:38.719081 2372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37467 } }
I20260927 10:54:38.719426 2372 leader_election.cc:302] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e8ba525583b4d5e9fe97dfcbd50ecf8; no voters:
I20260927 10:54:38.719661 2372 leader_election.cc:288] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:38.719766 2377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:38.719970 2377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e8ba525583b4d5e9fe97dfcbd50ecf8, State: Running, Role: LEADER
I20260927 10:54:38.720263 2377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37467 } }
I20260927 10:54:38.720788 2372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:38.721992 2378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37467 } } }
I20260927 10:54:38.722121 2378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:38.722406 2379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e8ba525583b4d5e9fe97dfcbd50ecf8. Latest consensus state: current_term: 1 leader_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8ba525583b4d5e9fe97dfcbd50ecf8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37467 } } }
I20260927 10:54:38.722504 2379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:38.722617 2386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:38.725322 2386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:38.730434 2386 catalog_manager.cc:1347] Generated new cluster ID: f11b3bcea13d432bab8fefff91c56d9f
I20260927 10:54:38.730511 2386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:38.757570 2386 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:38.758486 2386 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:38.768419 2386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8: Generated new TSK 0
I20260927 10:54:38.769181 2386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:38.805222 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37467
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:38.909885 2398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:38.910108 2398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:38.910148 2398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:38.913982 2398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:38.914029 2398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:38.914064 2398 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:54:38.914119 2398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:38.918980 2398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:38.919782 2398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:38.920665 2398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:38.929715 2403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:38.929750 2404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:38.929886 2406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:38.930017 2398 server_base.cc:1034] running on GCE node
I20260927 10:54:38.930429 2398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:38.931041 2398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:38.932232 2398 hybrid_clock.cc:648] HybridClock initialized: now 1790506478932213 us; error 33 us; skew 500 ppm
Sep 27 10:54:38 dist-test-slave-cffh krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:38.934880 2398 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:38.935724 2398 webserver.cc:460] Webserver started at http://127.1.46.193:38909/ using document root <none> and password file <none>
I20260927 10:54:38.936249 2398 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:38.936333 2398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:38.936548 2398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:38.938429 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "98c39dc869c6412481f57e280dd1166f"
format_stamp: "Formatted at 2026-09-27 10:54:38 on dist-test-slave-cffh"
server_key: "a446dc557e297f352e5e80fd5a063961"
server_key_iv: "50ba81378d261947a0cef149835975eb"
server_key_version: "encryptionkey@0"
I20260927 10:54:38.939018 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "98c39dc869c6412481f57e280dd1166f"
format_stamp: "Formatted at 2026-09-27 10:54:38 on dist-test-slave-cffh"
server_key: "a446dc557e297f352e5e80fd5a063961"
server_key_iv: "50ba81378d261947a0cef149835975eb"
server_key_version: "encryptionkey@0"
I20260927 10:54:38.942396 2398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:38.945026 2413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:38.946229 2398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:38.946398 2398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "98c39dc869c6412481f57e280dd1166f"
format_stamp: "Formatted at 2026-09-27 10:54:38 on dist-test-slave-cffh"
server_key: "a446dc557e297f352e5e80fd5a063961"
server_key_iv: "50ba81378d261947a0cef149835975eb"
server_key_version: "encryptionkey@0"
I20260927 10:54:38.946523 2398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:38.968742 2398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:38.972034 2398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:38.972251 2398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:38.973739 2398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:38.973814 2398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:38.973886 2398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:38.973948 2398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:54:38 dist-test-slave-cffh krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:38.985179 2398 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:45155
I20260927 10:54:38.985255 2528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:45155 every 8 connection(s)
I20260927 10:54:38.986202 2398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:38.987648 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:38.974346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:35503 (local address 127.1.46.254:37467)
0927 10:54:38.974497 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:38.974503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:38.975868 (+ 1365us) server_negotiation.cc:407] Connection header received
0927 10:54:38.977529 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:38.977536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:38.977594 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:38.977713 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:38.979023 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.979566 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.980745 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.981115 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.981233 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:38.983870 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:38.983874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:38.983878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:38.983908 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:38.985603 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:38.986081 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:38.986084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:38.986086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:38.986182 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:38.986549 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:38.986553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:38.986554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:38.986800 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:38.986908 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:38.987435 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 10:54:38.987532 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:54:38.988360 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:38.975208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37467 (local address 127.1.46.193:35503)
0927 10:54:38.975706 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:38.975765 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:38.977281 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:38.977886 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:38.977893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:38.978262 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:38.978884 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:38.978900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.979687 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.979692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:38.980626 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:38.980635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.981273 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.981278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:38.981437 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:38.981964 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:38.981983 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:38.983698 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:38.985706 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:38.985710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:38.985720 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:38.985963 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:38.986294 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:38.986299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:38.986304 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:38.986438 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:38.986962 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:38.986969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:38.987197 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 10:54:38.987352 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 10:54:38.987563 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
I20260927 10:54:38.989552 2529 heartbeater.cc:347] Connected to a master server at 127.1.46.254:37467
I20260927 10:54:38.989732 2529 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:38.990159 2529 heartbeater.cc:511] Master 127.1.46.254:37467 requested a full tablet report, sending...
I20260927 10:54:38.990690 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 2398
I20260927 10:54:38.990816 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:38.991130 1211 external_mini_cluster.cc:1428] Setting key 8e6cf67f5403551f0474aad7702c134b
I20260927 10:54:38.991767 2326 ts_manager.cc:194] Registered new tserver with Master: 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155)
I20260927 10:54:38.993109 2326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:35503
I20260927 10:54:38.999239 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:38.991618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37467 (local address 127.1.46.193:35727)
0927 10:54:38.992259 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:38.992282 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:38.992374 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:38.992611 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:38.992615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:38.992818 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:38.992959 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:38.992969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.993913 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.993919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:38.994670 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:38.994678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.995096 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.995099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:38.995189 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:38.995638 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:38.995649 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:38.995909 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:38.997748 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:38.997751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:38.997752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:38.997924 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:38.998224 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:38.998227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:38.998230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:38.998307 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:38.998734 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:38.998737 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:38.998824 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 10:54:38.998922 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:54:38.999040 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":17,"thread_start_us":85,"threads_started":1}
I20260927 10:54:38.999256 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:38.991692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:35727 (local address 127.1.46.254:37467)
0927 10:54:38.991792 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:38.991797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:38.992356 (+ 559us) server_negotiation.cc:407] Connection header received
0927 10:54:38.992451 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:38.992455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:38.992495 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:38.992573 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:38.993079 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.993759 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.994787 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:38.994989 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:38.995091 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:38.996040 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:38.996046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:38.996052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:38.996081 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:38.997655 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:38.998039 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:38.998043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:38.998046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:38.998126 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:38.998406 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:38.998411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:38.998414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:38.998562 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:38.998734 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:38.998982 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 10:54:38.999076 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:39.018070 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37467
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:39.120934 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:39.121155 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:39.121219 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:39.125298 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:39.125345 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:39.125391 2538 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:54:39.125453 2538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:39.130334 2538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:39.131253 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:39.132138 2538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:39.140933 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:39.140889 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:39.141160 2538 server_base.cc:1034] running on GCE node
W20260927 10:54:39.141182 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:39.141563 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:39.142170 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:39.143342 2538 hybrid_clock.cc:648] HybridClock initialized: now 1790506479143312 us; error 38 us; skew 500 ppm
Sep 27 10:54:39 dist-test-slave-cffh krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:39.150018 2538 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:39.151067 2538 webserver.cc:460] Webserver started at http://127.1.46.194:37381/ using document root <none> and password file <none>
I20260927 10:54:39.151611 2538 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:39.151677 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:39.151918 2538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:39.153908 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "67984452b75a4ef7b56e0d0879d1958e"
format_stamp: "Formatted at 2026-09-27 10:54:39 on dist-test-slave-cffh"
server_key: "d02d415c226f80502a170d32344df2e6"
server_key_iv: "aa2b75af0b748f420ddf4dbefd226cf7"
server_key_version: "encryptionkey@0"
I20260927 10:54:39.154475 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "67984452b75a4ef7b56e0d0879d1958e"
format_stamp: "Formatted at 2026-09-27 10:54:39 on dist-test-slave-cffh"
server_key: "d02d415c226f80502a170d32344df2e6"
server_key_iv: "aa2b75af0b748f420ddf4dbefd226cf7"
server_key_version: "encryptionkey@0"
I20260927 10:54:39.158097 2538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:54:39.160537 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:39.161834 2538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:39.161962 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "67984452b75a4ef7b56e0d0879d1958e"
format_stamp: "Formatted at 2026-09-27 10:54:39 on dist-test-slave-cffh"
server_key: "d02d415c226f80502a170d32344df2e6"
server_key_iv: "aa2b75af0b748f420ddf4dbefd226cf7"
server_key_version: "encryptionkey@0"
I20260927 10:54:39.162072 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:39.183029 2538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:39.185858 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:39.186055 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:39.187630 2538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:39.187680 2538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:39.187717 2538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:39.187770 2538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:54:39 dist-test-slave-cffh krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:39.198074 2538 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:33407
I20260927 10:54:39.198135 2668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:33407 every 8 connection(s)
I20260927 10:54:39.198937 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:39.201385 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.188384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:39775 (local address 127.1.46.254:37467)
0927 10:54:39.188538 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.188545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.189732 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 10:54:39.190954 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.190958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.191015 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.191127 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:39.192698 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.193544 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.195364 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.195551 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.195625 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.198083 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:39.198087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:39.198092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:39.198118 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:39.199758 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.200221 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.200223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.200225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.200290 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.200563 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.200565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.200566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.200727 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:39.200798 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.201187 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.201274 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:54:39.202049 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.189131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37467 (local address 127.1.46.194:39775)
0927 10:54:39.189579 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.189627 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.190715 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.191314 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.191322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.191815 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:39.192525 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.192549 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.193738 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.193745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.195243 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.195252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.195701 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.195707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.195886 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.196453 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:39.196471 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:39.197910 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:39.199874 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.199878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.199906 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.200135 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.200360 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.200362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.200364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.200491 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.200808 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:39.200812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:39.200995 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.201144 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.201344 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
I20260927 10:54:39.203382 2669 heartbeater.cc:347] Connected to a master server at 127.1.46.254:37467
I20260927 10:54:39.203543 2669 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:39.203572 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 2538
I20260927 10:54:39.203693 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:39.204078 1211 external_mini_cluster.cc:1428] Setting key fa076b760845aa7a003d27181e67d8cc
I20260927 10:54:39.204195 2669 heartbeater.cc:511] Master 127.1.46.254:37467 requested a full tablet report, sending...
I20260927 10:54:39.205199 2326 ts_manager.cc:194] Registered new tserver with Master: 67984452b75a4ef7b56e0d0879d1958e (127.1.46.194:33407)
I20260927 10:54:39.205922 2326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:39775
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:54:39.214475 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.205615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:46935 (local address 127.1.46.254:37467)
0927 10:54:39.205712 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.205717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.206892 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 10:54:39.206978 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.206981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.207026 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.207123 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:39.207724 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.208512 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.209803 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.210044 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.210119 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.211456 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:39.211458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:39.211460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:39.211479 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:39.212821 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.213322 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.213324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.213325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.213389 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.213700 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.213702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.213703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.213875 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:39.213946 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.214251 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.214365 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:54:39.214659 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.205425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37467 (local address 127.1.46.194:46935)
0927 10:54:39.206792 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.206809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.206902 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.207237 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.207241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.207467 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:39.207587 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.207596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.208657 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.208664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.209663 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.209677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.210162 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.210167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.210296 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.210795 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:39.210817 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:39.211324 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:39.212933 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.212937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.212939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.213221 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.213486 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.213492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.213497 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.213595 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.213975 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:39.213981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:39.214088 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.214209 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.214359 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:39.237250 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37467
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:39.342235 2678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:39.342442 2678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:39.342514 2678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:39.346426 2678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:39.346472 2678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:39.346508 2678 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:54:39.346537 2678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:39.351619 2678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:39.352520 2678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:39.353449 2678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:39.362177 2683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:39.362203 2686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:39.362159 2684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:39.362330 2678 server_base.cc:1034] running on GCE node
I20260927 10:54:39.362810 2678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:39.363412 2678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:39.364596 2678 hybrid_clock.cc:648] HybridClock initialized: now 1790506479364584 us; error 30 us; skew 500 ppm
Sep 27 10:54:39 dist-test-slave-cffh krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:39.367414 2678 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:39.368196 2678 webserver.cc:460] Webserver started at http://127.1.46.195:45421/ using document root <none> and password file <none>
I20260927 10:54:39.368709 2678 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:39.368822 2678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:39.369062 2678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:39.370949 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "5157d9ca7a694affa67c79685857071d"
format_stamp: "Formatted at 2026-09-27 10:54:39 on dist-test-slave-cffh"
server_key: "14c8a6ddc4d1d9ca80962616ec650200"
server_key_iv: "1c467255fdfb2d243ec47baedb7ee344"
server_key_version: "encryptionkey@0"
I20260927 10:54:39.371465 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "5157d9ca7a694affa67c79685857071d"
format_stamp: "Formatted at 2026-09-27 10:54:39 on dist-test-slave-cffh"
server_key: "14c8a6ddc4d1d9ca80962616ec650200"
server_key_iv: "1c467255fdfb2d243ec47baedb7ee344"
server_key_version: "encryptionkey@0"
I20260927 10:54:39.374763 2678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:39.376931 2693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:39.378039 2678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:39.378166 2678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "5157d9ca7a694affa67c79685857071d"
format_stamp: "Formatted at 2026-09-27 10:54:39 on dist-test-slave-cffh"
server_key: "14c8a6ddc4d1d9ca80962616ec650200"
server_key_iv: "1c467255fdfb2d243ec47baedb7ee344"
server_key_version: "encryptionkey@0"
I20260927 10:54:39.378268 2678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:39.390798 2678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:39.393646 2678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:39.393832 2678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:39.395198 2678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:39.395267 2678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:39.395331 2678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:39.395386 2678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:54:39 dist-test-slave-cffh krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:39.405582 2678 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:44733
I20260927 10:54:39.405781 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:44733 every 8 connection(s)
I20260927 10:54:39.406708 2678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:39.408733 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.395862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:53253 (local address 127.1.46.254:37467)
0927 10:54:39.396023 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.396028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.397181 (+ 1153us) server_negotiation.cc:407] Connection header received
0927 10:54:39.398468 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.398471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.398529 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.398667 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:39.400136 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.400690 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.402112 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.402421 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.402530 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.405046 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:39.405051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:39.405056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:39.405090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:39.406858 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.407440 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.407447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.407450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.407537 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.407857 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.407860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.407861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.408030 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:39.408122 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.408530 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.408617 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:54:39.409390 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.396585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37467 (local address 127.1.46.195:53253)
0927 10:54:39.397011 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.397069 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.398225 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.398840 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.398851 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.399340 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:39.399991 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.400015 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.400794 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.400797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.401984 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.401995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.402563 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.402622 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.402806 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.403297 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:39.403322 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:39.404888 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:39.406957 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.406961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.406970 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.407323 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.407635 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.407643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.407645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.407771 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.408138 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:39.408142 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:39.408346 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.408484 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.408710 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260927 10:54:39.410380 2809 heartbeater.cc:347] Connected to a master server at 127.1.46.254:37467
I20260927 10:54:39.410527 2809 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:39.410981 2809 heartbeater.cc:511] Master 127.1.46.254:37467 requested a full tablet report, sending...
I20260927 10:54:39.412006 2325 ts_manager.cc:194] Registered new tserver with Master: 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733)
I20260927 10:54:39.412588 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 2678
I20260927 10:54:39.412667 2325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:53253
I20260927 10:54:39.412707 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:39.413020 1211 external_mini_cluster.cc:1428] Setting key 3ee28cf7eefbf3e0aabc0c3cc64f282a
I20260927 10:54:39.414988 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:54:39.419795 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.412526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:44717 (local address 127.1.46.254:37467)
0927 10:54:39.412628 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.412631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.412959 (+ 328us) server_negotiation.cc:407] Connection header received
0927 10:54:39.413061 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.413066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.413110 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.413201 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:39.413692 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.414399 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.415410 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.415746 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.415858 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.416722 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:39.416728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:39.416733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:39.416764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:39.418153 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.418646 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.418650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.418653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.418731 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.419016 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.419018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.419019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.419239 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:39.419324 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.419614 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.419698 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:54:39.419900 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.412580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37467 (local address 127.1.46.195:44717)
0927 10:54:39.412889 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.412905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.412993 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.413215 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.413217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.413452 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:39.413572 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.413578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.414532 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.414536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.415299 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.415308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.415847 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.415849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.415911 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.416332 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:39.416342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:39.416590 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:39.418257 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.418263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.418268 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.418512 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.418818 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:39.418824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:39.418828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:39.418928 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:39.419345 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:39.419350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:39.419456 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.419568 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.419713 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
Sep 27 10:54:39 dist-test-slave-cffh krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506479, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:54:39 dist-test-slave-cffh krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506479, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:39.438740 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.431860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.1.46.254:37467)
0927 10:54:39.431969 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.431972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.431982 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:54:39.432009 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.432011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.432041 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.432102 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:39.432739 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.433200 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.433937 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.434128 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.434198 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.436278 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:39.436281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:39.436283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:39.436303 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:39.437430 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.437782 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.437784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.437786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.437850 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:39.438061 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:39.438063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:39.438064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:39.438200 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:39.438294 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.438481 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.438558 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:54:39.441090 2325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:39.444139 2325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:39.457283 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.452435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44733 (local address 127.0.0.1:54410)
0927 10:54:39.452776 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.452824 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.452945 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.453647 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.453650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.453680 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:39.453809 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.453814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.454857 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.454863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.456019 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.456027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.456805 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.456809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.456908 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.456946 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.457049 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.457150 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"spinlock_wait_cycles":7296,"thread_start_us":73,"threads_started":1}
I20260927 10:54:39.457298 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.452575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.1.46.195:44733)
0927 10:54:39.453047 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.453058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.453079 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:54:39.453148 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.453155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.453347 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.453495 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:39.453928 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.454753 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.456141 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.456703 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.456809 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.456953 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.457088 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.457170 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":88,"threads_started":1}
I20260927 10:54:39.458177 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.451779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:33407 (local address 127.0.0.1:55406)
0927 10:54:39.452776 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.452824 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.452945 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.453497 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.453502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.453522 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:39.453671 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.453677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.455336 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.455339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.456578 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.456588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.457805 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.457809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.457892 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.457915 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.457980 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.458054 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":65,"spinlock_wait_cycles":1792,"thread_start_us":63,"threads_started":1}
I20260927 10:54:39.458307 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.451909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.1.46.194:33407)
0927 10:54:39.452236 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.452246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.453084 (+ 838us) server_negotiation.cc:407] Connection header received
0927 10:54:39.453180 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.453185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.453357 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.453507 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:39.453815 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.455055 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.456718 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.457688 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.457805 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.457950 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.458001 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.458085 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":126,"threads_started":1}
I20260927 10:54:39.458314 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.451972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.1.46.193:45155)
0927 10:54:39.452321 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.452339 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.453205 (+ 866us) server_negotiation.cc:407] Connection header received
0927 10:54:39.453277 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.453281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.453440 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.453555 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:39.454020 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.455114 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.456703 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.457508 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.457631 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.457772 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.458079 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.458182 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":69,"threads_started":1}
I20260927 10:54:39.458689 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.451783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45155 (local address 127.0.0.1:44694)
0927 10:54:39.453101 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.453114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.453189 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.453674 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.453677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.453691 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:39.453842 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.453848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.455234 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.455239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.456581 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.456588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.457832 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.457835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.457918 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.457940 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.458474 (+ 534us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.458539 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":726,"spinlock_wait_cycles":917760,"thread_start_us":87,"threads_started":1}
I20260927 10:54:39.460366 2743 tablet_service.cc:1467] Processing CreateTablet for tablet 550ba40b56cb47b3abf91e472ec725ee (DEFAULT_TABLE table=test-table [id=9d8615d9e2894d6db3f5e43c9ce551b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:39.460780 2463 tablet_service.cc:1467] Processing CreateTablet for tablet 550ba40b56cb47b3abf91e472ec725ee (DEFAULT_TABLE table=test-table [id=9d8615d9e2894d6db3f5e43c9ce551b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:39.460780 2603 tablet_service.cc:1467] Processing CreateTablet for tablet 550ba40b56cb47b3abf91e472ec725ee (DEFAULT_TABLE table=test-table [id=9d8615d9e2894d6db3f5e43c9ce551b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:39.461270 2743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550ba40b56cb47b3abf91e472ec725ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:39.461721 2603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550ba40b56cb47b3abf91e472ec725ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:39.461721 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550ba40b56cb47b3abf91e472ec725ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:39.468024 2832 tablet_bootstrap.cc:492] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Bootstrap starting.
I20260927 10:54:39.469223 2833 tablet_bootstrap.cc:492] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Bootstrap starting.
I20260927 10:54:39.470860 2832 tablet_bootstrap.cc:654] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:39.470858 2834 tablet_bootstrap.cc:492] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Bootstrap starting.
I20260927 10:54:39.471128 2833 tablet_bootstrap.cc:654] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:39.471709 2832 log.cc:824] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:39.471820 2833 log.cc:824] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:39.472731 2834 tablet_bootstrap.cc:654] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:39.473409 2833 tablet_bootstrap.cc:492] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: No bootstrap required, opened a new log
I20260927 10:54:39.473528 2834 log.cc:824] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:39.473570 2832 tablet_bootstrap.cc:492] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: No bootstrap required, opened a new log
I20260927 10:54:39.473605 2833 ts_tablet_manager.cc:1397] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:54:39.473727 2832 ts_tablet_manager.cc:1397] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:54:39.475294 2834 tablet_bootstrap.cc:492] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: No bootstrap required, opened a new log
I20260927 10:54:39.475461 2834 ts_tablet_manager.cc:1397] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:54:39.476738 2833 raft_consensus.cc:348] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.476949 2833 raft_consensus.cc:374] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:39.477005 2833 raft_consensus.cc:729] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c39dc869c6412481f57e280dd1166f, State: Initialized, Role: FOLLOWER
I20260927 10:54:39.477420 2833 consensus_queue.cc:260] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.477478 2832 raft_consensus.cc:348] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.477721 2832 raft_consensus.cc:374] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:39.477774 2832 raft_consensus.cc:729] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5157d9ca7a694affa67c79685857071d, State: Initialized, Role: FOLLOWER
I20260927 10:54:39.478199 2832 consensus_queue.cc:260] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.478343 2833 ts_tablet_manager.cc:1428] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260927 10:54:39.478701 2529 heartbeater.cc:503] Master 127.1.46.254:37467 was elected leader, sending a full tablet report...
I20260927 10:54:39.478785 2809 heartbeater.cc:503] Master 127.1.46.254:37467 was elected leader, sending a full tablet report...
I20260927 10:54:39.479120 2832 ts_tablet_manager.cc:1428] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:54:39.479053 2834 raft_consensus.cc:348] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.479290 2834 raft_consensus.cc:374] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:39.479341 2834 raft_consensus.cc:729] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67984452b75a4ef7b56e0d0879d1958e, State: Initialized, Role: FOLLOWER
I20260927 10:54:39.479880 2834 consensus_queue.cc:260] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.480422 2669 heartbeater.cc:503] Master 127.1.46.254:37467 was elected leader, sending a full tablet report...
I20260927 10:54:39.480736 2834 ts_tablet_manager.cc:1428] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:54:39.487279 2530 tablet.cc:2369] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:54:39.657847 2810 tablet.cc:2369] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:54:39.700129 2670 tablet.cc:2369] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:54:39.738472 2840 raft_consensus.cc:482] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:39.738787 2840 raft_consensus.cc:504] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.740119 2840 leader_election.cc:288] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155), 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733)
I20260927 10:54:39.744189 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.740387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:44139 (local address 127.1.46.193:45155)
0927 10:54:39.740513 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.740517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.740717 (+ 200us) server_negotiation.cc:407] Connection header received
0927 10:54:39.740869 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.740874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.740933 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.741000 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:39.741348 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.741878 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.743125 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.743541 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.743672 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.743724 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.743986 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.744058 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:54:39.744201 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.740526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:50179 (local address 127.1.46.195:44733)
0927 10:54:39.740636 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.740640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.740790 (+ 150us) server_negotiation.cc:407] Connection header received
0927 10:54:39.740939 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.740942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.741011 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.741115 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:39.741376 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.741895 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.743136 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.743543 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.743695 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.743758 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.743990 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.744058 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 10:54:39.744226 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.740352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44733 (local address 127.1.46.194:50179)
0927 10:54:39.740669 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.740687 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.740817 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.741140 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.741142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.741156 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:39.741243 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.741248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.742031 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.742034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.742998 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.743007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.743695 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.743698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.743810 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.743882 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.743971 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.744042 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":117,"thread_start_us":72,"threads_started":1}
I20260927 10:54:39.744235 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.740277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45155 (local address 127.1.46.194:44139)
0927 10:54:39.740589 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:39.740615 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:39.740721 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:39.741036 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:39.741042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:39.741064 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:39.741217 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.741225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.741992 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.741999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.742992 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:39.743001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.743676 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.743680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:39.743841 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.743876 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:54:39.743973 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:54:39.744060 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":69,"thread_start_us":72,"threads_started":1}
I20260927 10:54:39.744928 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ba40b56cb47b3abf91e472ec725ee" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c39dc869c6412481f57e280dd1166f" is_pre_election: true
I20260927 10:54:39.744987 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ba40b56cb47b3abf91e472ec725ee" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5157d9ca7a694affa67c79685857071d" is_pre_election: true
I20260927 10:54:39.745222 2483 raft_consensus.cc:2454] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 0.
I20260927 10:54:39.745255 2759 raft_consensus.cc:2454] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 0.
I20260927 10:54:39.745757 2554 leader_election.cc:302] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5157d9ca7a694affa67c79685857071d, 67984452b75a4ef7b56e0d0879d1958e; no voters:
I20260927 10:54:39.746042 2840 raft_consensus.cc:2781] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:39.746145 2840 raft_consensus.cc:482] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:39.746212 2840 raft_consensus.cc:3037] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:39.747361 2840 raft_consensus.cc:504] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.747756 2840 leader_election.cc:288] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 election: Requested vote from peers 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155), 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733)
I20260927 10:54:39.748080 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ba40b56cb47b3abf91e472ec725ee" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c39dc869c6412481f57e280dd1166f"
I20260927 10:54:39.748179 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ba40b56cb47b3abf91e472ec725ee" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5157d9ca7a694affa67c79685857071d"
I20260927 10:54:39.748245 2483 raft_consensus.cc:3037] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:39.748353 2759 raft_consensus.cc:3037] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:39.749270 2483 raft_consensus.cc:2454] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 1.
I20260927 10:54:39.749581 2557 leader_election.cc:302] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67984452b75a4ef7b56e0d0879d1958e, 98c39dc869c6412481f57e280dd1166f; no voters:
I20260927 10:54:39.749758 2840 raft_consensus.cc:2781] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:39.749953 2840 raft_consensus.cc:686] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 1 LEADER]: Becoming Leader. State: Replica: 67984452b75a4ef7b56e0d0879d1958e, State: Running, Role: LEADER
I20260927 10:54:39.750092 2759 raft_consensus.cc:2454] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 1.
I20260927 10:54:39.750275 2840 consensus_queue.cc:237] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.753473 2325 catalog_manager.cc:5495] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e reported cstate change: term changed from 0 to 1, leader changed from <none> to 67984452b75a4ef7b56e0d0879d1958e (127.1.46.194). New cstate: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:39.823482 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:39.819599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.1.46.194:33407)
0927 10:54:39.819737 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:39.819743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:39.819775 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:54:39.820003 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:39.820006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:39.820062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:39.820161 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:39.820410 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.820940 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.821967 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:39.822193 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:39.822311 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:39.822655 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:39.823049 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:39.823181 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:39.823260 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 10:54:39.823316 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:54:39.828949 2483 raft_consensus.cc:1261] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Refusing update from remote peer 67984452b75a4ef7b56e0d0879d1958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:39.828985 2759 raft_consensus.cc:1261] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Refusing update from remote peer 67984452b75a4ef7b56e0d0879d1958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:39.829614 2840 consensus_queue.cc:1035] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:39.829789 2843 consensus_queue.cc:1035] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:39.836988 2848 mvcc.cc:205] Tried to move back new op lower bound from 7333914541372260352 to 7333914541059944448. Current Snapshot: MvccSnapshot[applied={T|T < 7333914541372260352}]
I20260927 10:54:39.848547 2325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:39.848718 2325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:39.851238 2325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8: Sending DeleteTablet for 3 replicas of tablet 550ba40b56cb47b3abf91e472ec725ee
I20260927 10:54:39.852494 2743 tablet_service.cc:1514] Processing DeleteTablet for tablet 550ba40b56cb47b3abf91e472ec725ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:39 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:54410
I20260927 10:54:39.852494 2603 tablet_service.cc:1514] Processing DeleteTablet for tablet 550ba40b56cb47b3abf91e472ec725ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:39 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:55406
I20260927 10:54:39.852509 2463 tablet_service.cc:1514] Processing DeleteTablet for tablet 550ba40b56cb47b3abf91e472ec725ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:39 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44694
I20260927 10:54:39.853030 2858 tablet_replica.cc:330] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: stopping tablet replica
I20260927 10:54:39.853325 2858 raft_consensus.cc:2229] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:39.853740 2858 raft_consensus.cc:2258] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:39.854001 2325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:54:39.854104 2857 tablet_replica.cc:330] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: stopping tablet replica
I20260927 10:54:39.854367 2857 raft_consensus.cc:2229] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 10:54:39.854400 2325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:39.854709 2857 raft_consensus.cc:2258] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:39.855212 2858 ts_tablet_manager.cc:1905] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:39.856665 2857 ts_tablet_manager.cc:1905] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:39.857913 2858 ts_tablet_manager.cc:1918] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:39.858008 2858 log.cc:1196] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/550ba40b56cb47b3abf91e472ec725ee
I20260927 10:54:39.858335 2858 ts_tablet_manager.cc:1939] T 550ba40b56cb47b3abf91e472ec725ee P 67984452b75a4ef7b56e0d0879d1958e: Deleting consensus metadata
I20260927 10:54:39.858773 2856 tablet_replica.cc:330] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: stopping tablet replica
I20260927 10:54:39.859061 2856 raft_consensus.cc:2229] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:39.859081 2603 tablet_service.cc:1467] Processing CreateTablet for tablet c7a69ec523ca4b58865b3003926652af (DEFAULT_TABLE table=test-table [id=7996d607f2424ffeb69e7f49cf840299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:39.859331 2603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a69ec523ca4b58865b3003926652af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:39.859323 2856 raft_consensus.cc:2258] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:39.859766 2310 catalog_manager.cc:4841] TS 67984452b75a4ef7b56e0d0879d1958e (127.1.46.194:33407): tablet 550ba40b56cb47b3abf91e472ec725ee (table test-table [id=9d8615d9e2894d6db3f5e43c9ce551b1]) successfully deleted
I20260927 10:54:39.860046 2743 tablet_service.cc:1467] Processing CreateTablet for tablet c7a69ec523ca4b58865b3003926652af (DEFAULT_TABLE table=test-table [id=7996d607f2424ffeb69e7f49cf840299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:39.860245 2857 ts_tablet_manager.cc:1918] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:39.860240 2463 tablet_service.cc:1467] Processing CreateTablet for tablet c7a69ec523ca4b58865b3003926652af (DEFAULT_TABLE table=test-table [id=7996d607f2424ffeb69e7f49cf840299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:39.860296 2743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a69ec523ca4b58865b3003926652af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:39.860320 2857 log.cc:1196] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/550ba40b56cb47b3abf91e472ec725ee
I20260927 10:54:39.860468 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a69ec523ca4b58865b3003926652af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:39.860635 2857 ts_tablet_manager.cc:1939] T 550ba40b56cb47b3abf91e472ec725ee P 5157d9ca7a694affa67c79685857071d: Deleting consensus metadata
I20260927 10:54:39.861490 2834 tablet_bootstrap.cc:492] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: Bootstrap starting.
I20260927 10:54:39.862408 2834 tablet_bootstrap.cc:654] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:39.862449 2832 tablet_bootstrap.cc:492] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: Bootstrap starting.
I20260927 10:54:39.862766 2856 ts_tablet_manager.cc:1905] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:39.863312 2832 tablet_bootstrap.cc:654] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:39.863900 2833 tablet_bootstrap.cc:492] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: Bootstrap starting.
I20260927 10:54:39.864796 2833 tablet_bootstrap.cc:654] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:39.865366 2856 ts_tablet_manager.cc:1918] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:39.865456 2856 log.cc:1196] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/550ba40b56cb47b3abf91e472ec725ee
I20260927 10:54:39.865773 2856 ts_tablet_manager.cc:1939] T 550ba40b56cb47b3abf91e472ec725ee P 98c39dc869c6412481f57e280dd1166f: Deleting consensus metadata
I20260927 10:54:39.866143 2310 catalog_manager.cc:4841] TS 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733): tablet 550ba40b56cb47b3abf91e472ec725ee (table test-table [id=9d8615d9e2894d6db3f5e43c9ce551b1]) successfully deleted
I20260927 10:54:39.866735 2832 tablet_bootstrap.cc:492] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: No bootstrap required, opened a new log
I20260927 10:54:39.866825 2832 ts_tablet_manager.cc:1397] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:54:39.866945 2313 catalog_manager.cc:4841] TS 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155): tablet 550ba40b56cb47b3abf91e472ec725ee (table test-table [id=9d8615d9e2894d6db3f5e43c9ce551b1]) successfully deleted
I20260927 10:54:39.867282 2832 raft_consensus.cc:348] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.867389 2832 raft_consensus.cc:374] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:39.867411 2832 raft_consensus.cc:729] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5157d9ca7a694affa67c79685857071d, State: Initialized, Role: FOLLOWER
I20260927 10:54:39.867568 2832 consensus_queue.cc:260] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.867825 2832 ts_tablet_manager.cc:1428] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:54:39.868249 2833 tablet_bootstrap.cc:492] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: No bootstrap required, opened a new log
I20260927 10:54:39.868350 2833 ts_tablet_manager.cc:1397] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 10:54:39.868719 2833 raft_consensus.cc:348] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.868829 2833 raft_consensus.cc:374] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:39.868871 2833 raft_consensus.cc:729] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c39dc869c6412481f57e280dd1166f, State: Initialized, Role: FOLLOWER
I20260927 10:54:39.869004 2833 consensus_queue.cc:260] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.869256 2833 ts_tablet_manager.cc:1428] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:54:39.871479 2834 tablet_bootstrap.cc:492] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: No bootstrap required, opened a new log
I20260927 10:54:39.871569 2834 ts_tablet_manager.cc:1397] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 10:54:39.872030 2834 raft_consensus.cc:348] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.872162 2834 raft_consensus.cc:374] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:39.872206 2834 raft_consensus.cc:729] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67984452b75a4ef7b56e0d0879d1958e, State: Initialized, Role: FOLLOWER
I20260927 10:54:39.872329 2834 consensus_queue.cc:260] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:39.872597 2834 ts_tablet_manager.cc:1428] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:40.046242 2843 raft_consensus.cc:482] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:40.046368 2843 raft_consensus.cc:504] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.046860 2843 leader_election.cc:288] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155), 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733)
I20260927 10:54:40.047288 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a69ec523ca4b58865b3003926652af" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c39dc869c6412481f57e280dd1166f" is_pre_election: true
I20260927 10:54:40.047334 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a69ec523ca4b58865b3003926652af" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5157d9ca7a694affa67c79685857071d" is_pre_election: true
I20260927 10:54:40.047439 2483 raft_consensus.cc:2454] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 0.
I20260927 10:54:40.047513 2759 raft_consensus.cc:2454] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 0.
I20260927 10:54:40.047809 2557 leader_election.cc:302] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67984452b75a4ef7b56e0d0879d1958e, 98c39dc869c6412481f57e280dd1166f; no voters:
I20260927 10:54:40.048048 2843 raft_consensus.cc:2781] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:40.048120 2843 raft_consensus.cc:482] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:40.048174 2843 raft_consensus.cc:3037] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:40.049103 2843 raft_consensus.cc:504] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.049482 2843 leader_election.cc:288] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 election: Requested vote from peers 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155), 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733)
I20260927 10:54:40.049784 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a69ec523ca4b58865b3003926652af" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5157d9ca7a694affa67c79685857071d"
I20260927 10:54:40.049809 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a69ec523ca4b58865b3003926652af" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c39dc869c6412481f57e280dd1166f"
I20260927 10:54:40.049928 2759 raft_consensus.cc:3037] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:40.049943 2483 raft_consensus.cc:3037] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:40.050921 2759 raft_consensus.cc:2454] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 1.
I20260927 10:54:40.050948 2483 raft_consensus.cc:2454] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 1.
I20260927 10:54:40.051177 2554 leader_election.cc:302] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5157d9ca7a694affa67c79685857071d, 67984452b75a4ef7b56e0d0879d1958e; no voters:
I20260927 10:54:40.051334 2843 raft_consensus.cc:2781] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:40.051419 2843 raft_consensus.cc:686] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 1 LEADER]: Becoming Leader. State: Replica: 67984452b75a4ef7b56e0d0879d1958e, State: Running, Role: LEADER
I20260927 10:54:40.051563 2843 consensus_queue.cc:237] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.053022 2325 catalog_manager.cc:5495] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e reported cstate change: term changed from 0 to 1, leader changed from <none> to 67984452b75a4ef7b56e0d0879d1958e (127.1.46.194). New cstate: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:54:40 dist-test-slave-cffh krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:40.119807 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.112075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37467 (local address 127.0.0.1:52048)
0927 10:54:40.112334 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:40.112346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:40.112422 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:40.112659 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:40.112661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:40.113042 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:40.113124 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:40.113130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.113840 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.113843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:40.114370 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:40.114377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.114787 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.114790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:40.114848 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.115328 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:40.115339 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:40.116822 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:40.118355 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:40.118362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:40.118368 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:40.118694 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:40.118928 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:40.118930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:40.118931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:40.119007 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:40.119316 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:40.119321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:40.119411 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 10:54:40.119498 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:54:40.119646 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260927 10:54:40.119825 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.112107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.1.46.254:37467)
0927 10:54:40.112370 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:40.112376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:40.112389 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:40.112499 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:40.112501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:40.112536 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:40.112606 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:40.113224 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.113749 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.114465 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.114695 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.114768 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.116951 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:40.116954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:40.116956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:40.116975 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:40.118258 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:40.118781 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:40.118783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:40.118784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:40.118855 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:40.119077 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:40.119079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:40.119080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:40.119234 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:40.119304 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:40.119547 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:54:40.119705 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":190,"thread_start_us":112,"threads_started":1}
I20260927 10:54:40.123657 2325 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:52048:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:54:40.124383 2325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:40.128896 2463 tablet_service.cc:1467] Processing CreateTablet for tablet 7153eaeca2f74e8689e1241036cd0c30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7e3c1b5b12447b3bc4fbc92696b9491]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:54:40.128952 2603 tablet_service.cc:1467] Processing CreateTablet for tablet 7153eaeca2f74e8689e1241036cd0c30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7e3c1b5b12447b3bc4fbc92696b9491]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:54:40.129386 2603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7153eaeca2f74e8689e1241036cd0c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:40.129544 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7153eaeca2f74e8689e1241036cd0c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:40.129864 2743 tablet_service.cc:1467] Processing CreateTablet for tablet 7153eaeca2f74e8689e1241036cd0c30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7e3c1b5b12447b3bc4fbc92696b9491]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:54:40.130152 2743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7153eaeca2f74e8689e1241036cd0c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:40.132313 2834 tablet_bootstrap.cc:492] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: Bootstrap starting.
I20260927 10:54:40.132653 2833 tablet_bootstrap.cc:492] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: Bootstrap starting.
I20260927 10:54:40.132936 2832 tablet_bootstrap.cc:492] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: Bootstrap starting.
I20260927 10:54:40.133239 2834 tablet_bootstrap.cc:654] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:40.133630 2833 tablet_bootstrap.cc:654] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:40.133783 2832 tablet_bootstrap.cc:654] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:40.134814 2832 tablet_bootstrap.cc:492] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: No bootstrap required, opened a new log
I20260927 10:54:40.134842 2833 tablet_bootstrap.cc:492] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: No bootstrap required, opened a new log
I20260927 10:54:40.134912 2832 ts_tablet_manager.cc:1397] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:40.134931 2833 ts_tablet_manager.cc:1397] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:54:40.135291 2832 raft_consensus.cc:348] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.135315 2833 raft_consensus.cc:348] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.135417 2832 raft_consensus.cc:374] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:40.135413 2833 raft_consensus.cc:374] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:40.135462 2832 raft_consensus.cc:729] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5157d9ca7a694affa67c79685857071d, State: Initialized, Role: FOLLOWER
I20260927 10:54:40.135468 2833 raft_consensus.cc:729] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c39dc869c6412481f57e280dd1166f, State: Initialized, Role: FOLLOWER
I20260927 10:54:40.135594 2832 consensus_queue.cc:260] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.135608 2833 consensus_queue.cc:260] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.135839 2839 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.135903 2833 ts_tablet_manager.cc:1428] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:40.135957 2839 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:40.136152 2838 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.136279 2838 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:40.136305 2832 ts_tablet_manager.cc:1428] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:54:40.137459 2834 tablet_bootstrap.cc:492] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: No bootstrap required, opened a new log
I20260927 10:54:40.137568 2834 ts_tablet_manager.cc:1397] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 10:54:40.138002 2834 raft_consensus.cc:348] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.138140 2834 raft_consensus.cc:374] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:40.138190 2834 raft_consensus.cc:729] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67984452b75a4ef7b56e0d0879d1958e, State: Initialized, Role: FOLLOWER
I20260927 10:54:40.138335 2834 consensus_queue.cc:260] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.138474 2843 raft_consensus.cc:482] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:40.138569 2843 raft_consensus.cc:504] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.138985 2843 leader_election.cc:288] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155), 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733)
I20260927 10:54:40.139182 2840 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.139307 2840 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:40.139324 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7153eaeca2f74e8689e1241036cd0c30" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5157d9ca7a694affa67c79685857071d" is_pre_election: true
I20260927 10:54:40.139446 2759 raft_consensus.cc:2454] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 0.
I20260927 10:54:40.139453 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7153eaeca2f74e8689e1241036cd0c30" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c39dc869c6412481f57e280dd1166f" is_pre_election: true
I20260927 10:54:40.139552 2483 raft_consensus.cc:2454] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 0.
I20260927 10:54:40.139719 2554 leader_election.cc:302] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5157d9ca7a694affa67c79685857071d, 67984452b75a4ef7b56e0d0879d1958e; no voters:
I20260927 10:54:40.140020 2840 raft_consensus.cc:2781] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:40.140077 2840 raft_consensus.cc:482] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:40.140117 2840 raft_consensus.cc:3037] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:40.140829 2840 raft_consensus.cc:504] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.141089 2840 leader_election.cc:288] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 election: Requested vote from peers 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155), 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733)
I20260927 10:54:40.141350 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7153eaeca2f74e8689e1241036cd0c30" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c39dc869c6412481f57e280dd1166f"
I20260927 10:54:40.141474 2483 raft_consensus.cc:3037] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:40.138656 2834 ts_tablet_manager.cc:1428] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 10:54:40.142505 2483 raft_consensus.cc:2454] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 1.
I20260927 10:54:40.142755 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7153eaeca2f74e8689e1241036cd0c30" candidate_uuid: "67984452b75a4ef7b56e0d0879d1958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5157d9ca7a694affa67c79685857071d"
I20260927 10:54:40.142812 2557 leader_election.cc:302] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67984452b75a4ef7b56e0d0879d1958e, 98c39dc869c6412481f57e280dd1166f; no voters:
I20260927 10:54:40.142859 2759 raft_consensus.cc:3037] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:40.143136 2840 raft_consensus.cc:2781] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:40.143241 2840 raft_consensus.cc:686] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [term 1 LEADER]: Becoming Leader. State: Replica: 67984452b75a4ef7b56e0d0879d1958e, State: Running, Role: LEADER
I20260927 10:54:40.143378 2840 consensus_queue.cc:237] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } }
I20260927 10:54:40.143873 2759 raft_consensus.cc:2454] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67984452b75a4ef7b56e0d0879d1958e in term 1.
I20260927 10:54:40.143980 2840 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: TxnStatusTablet state changed. Reason: New leader 67984452b75a4ef7b56e0d0879d1958e. Latest consensus state: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.144073 2840 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:54:40.144341 2874 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:54:40.144506 2874 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:54:40.144886 2326 catalog_manager.cc:5495] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e reported cstate change: term changed from 0 to 1, leader changed from <none> to 67984452b75a4ef7b56e0d0879d1958e (127.1.46.194). New cstate: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:40.156383 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.153172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.1.46.194:33407)
0927 10:54:40.153288 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:40.153292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:40.153421 (+ 129us) server_negotiation.cc:407] Connection header received
0927 10:54:40.153599 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:40.153605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:40.153686 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:40.153768 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:40.154019 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.154510 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.155354 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.155580 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.155685 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.155948 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:40.156020 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:40.156100 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:40.156219 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 10:54:40.156263 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:54:40.156383 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.153075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:33407 (local address 127.0.0.1:55420)
0927 10:54:40.153335 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:40.153349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:40.153484 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:40.153802 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:40.153805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:40.153814 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:54:40.153921 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:40.153925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.154647 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.154651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:40.155240 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:40.155247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.155683 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.155686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:40.155776 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.155825 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:54:40.156121 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:54:40.156135 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 10:54:40.156199 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 10:54:40.156277 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260927 10:54:40.158563 2759 raft_consensus.cc:1261] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Refusing update from remote peer 67984452b75a4ef7b56e0d0879d1958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:40.158564 2483 raft_consensus.cc:1261] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Refusing update from remote peer 67984452b75a4ef7b56e0d0879d1958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:40.158990 2843 consensus_queue.cc:1035] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:40.159178 2840 consensus_queue.cc:1035] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:40.160421 2838 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: TxnStatusTablet state changed. Reason: New leader 67984452b75a4ef7b56e0d0879d1958e. Latest consensus state: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.160434 2839 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: TxnStatusTablet state changed. Reason: New leader 67984452b75a4ef7b56e0d0879d1958e. Latest consensus state: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.160526 2839 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:40.160526 2838 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:40.161677 2843 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.161769 2843 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:54:40.162002 2853 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.162110 2853 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 67984452b75a4ef7b56e0d0879d1958e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:54:40.162078 2838 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.162174 2838 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 98c39dc869c6412481f57e280dd1166f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:40.162276 2846 mvcc.cc:205] Tried to move back new op lower bound from 7333914542725976064 to 7333914542668337152. Current Snapshot: MvccSnapshot[applied={T|T < 7333914542725976064 or (T in {7333914542725976064})}]
I20260927 10:54:40.164312 2839 tablet_replica.cc:439] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67984452b75a4ef7b56e0d0879d1958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 } } peers { permanent_uuid: "67984452b75a4ef7b56e0d0879d1958e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 33407 } } peers { permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 } } }
I20260927 10:54:40.164438 2839 tablet_replica.cc:442] T 7153eaeca2f74e8689e1241036cd0c30 P 5157d9ca7a694affa67c79685857071d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:40.176069 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.172509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:56069 (local address 127.1.46.194:33407)
0927 10:54:40.172637 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:40.172641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:40.172902 (+ 261us) server_negotiation.cc:407] Connection header received
0927 10:54:40.173000 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:40.173002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:40.173041 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:40.173106 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:40.173349 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.173865 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.175063 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.175281 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.175363 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.175672 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:40.175733 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:40.175789 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:40.175908 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 10:54:40.175946 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:54:40.176122 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.172401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:33407 (local address 127.1.46.194:56069)
0927 10:54:40.172788 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:40.172802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:40.172902 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:40.173142 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:40.173145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:40.173152 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:54:40.173249 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:40.173254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.173971 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.173975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:40.174945 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:40.174955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.175393 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.175397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:40.175480 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.175526 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:54:40.175821 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:54:40.175827 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:54:40.175892 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:54:40.175969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":124,"threads_started":1}
I20260927 10:54:40.182159 2759 raft_consensus.cc:1261] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Refusing update from remote peer 67984452b75a4ef7b56e0d0879d1958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:40.182587 2483 raft_consensus.cc:1261] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Refusing update from remote peer 67984452b75a4ef7b56e0d0879d1958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:40.182566 2853 consensus_queue.cc:1035] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5157d9ca7a694affa67c79685857071d" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:40.182962 2853 consensus_queue.cc:1035] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c39dc869c6412481f57e280dd1166f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:40.187243 2847 mvcc.cc:205] Tried to move back new op lower bound from 7333914542823104512 to 7333914542292213760. Current Snapshot: MvccSnapshot[applied={T|T < 7333914542823104512}]
I20260927 10:54:40.218554 2325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:40.218767 2325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:40.220638 2325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e8ba525583b4d5e9fe97dfcbd50ecf8: Sending DeleteTablet for 3 replicas of tablet c7a69ec523ca4b58865b3003926652af
I20260927 10:54:40.221223 2463 tablet_service.cc:1514] Processing DeleteTablet for tablet c7a69ec523ca4b58865b3003926652af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:40 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44694
I20260927 10:54:40.221401 2856 tablet_replica.cc:330] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: stopping tablet replica
I20260927 10:54:40.221537 2856 raft_consensus.cc:2229] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:40.221648 2856 raft_consensus.cc:2258] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:40.221933 2743 tablet_service.cc:1514] Processing DeleteTablet for tablet c7a69ec523ca4b58865b3003926652af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:40 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:54410
I20260927 10:54:40.222093 2857 tablet_replica.cc:330] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: stopping tablet replica
I20260927 10:54:40.222210 2857 raft_consensus.cc:2229] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:40.222349 2857 raft_consensus.cc:2258] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:40.222488 2856 ts_tablet_manager.cc:1905] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:40.222512 2603 tablet_service.cc:1514] Processing DeleteTablet for tablet c7a69ec523ca4b58865b3003926652af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:40 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:55406
I20260927 10:54:40.222683 2858 tablet_replica.cc:330] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: stopping tablet replica
I20260927 10:54:40.222771 2858 raft_consensus.cc:2229] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:40.222914 2858 raft_consensus.cc:2258] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:40.223024 2857 ts_tablet_manager.cc:1905] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:40.223591 2858 ts_tablet_manager.cc:1905] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:40.225008 2856 ts_tablet_manager.cc:1918] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:40.225059 2856 log.cc:1196] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/c7a69ec523ca4b58865b3003926652af
I20260927 10:54:40.225243 2856 ts_tablet_manager.cc:1939] T c7a69ec523ca4b58865b3003926652af P 98c39dc869c6412481f57e280dd1166f: Deleting consensus metadata
I20260927 10:54:40.225737 2857 ts_tablet_manager.cc:1918] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:40.225809 2857 log.cc:1196] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/c7a69ec523ca4b58865b3003926652af
I20260927 10:54:40.225790 2313 catalog_manager.cc:4841] TS 98c39dc869c6412481f57e280dd1166f (127.1.46.193:45155): tablet c7a69ec523ca4b58865b3003926652af (table test-table [id=7996d607f2424ffeb69e7f49cf840299]) successfully deleted
I20260927 10:54:40.226075 2857 ts_tablet_manager.cc:1939] T c7a69ec523ca4b58865b3003926652af P 5157d9ca7a694affa67c79685857071d: Deleting consensus metadata
I20260927 10:54:40.226217 2858 ts_tablet_manager.cc:1918] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:40.226289 2858 log.cc:1196] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/c7a69ec523ca4b58865b3003926652af
I20260927 10:54:40.226529 2858 ts_tablet_manager.cc:1939] T c7a69ec523ca4b58865b3003926652af P 67984452b75a4ef7b56e0d0879d1958e: Deleting consensus metadata
I20260927 10:54:40.226809 2310 catalog_manager.cc:4841] TS 5157d9ca7a694affa67c79685857071d (127.1.46.195:44733): tablet c7a69ec523ca4b58865b3003926652af (table test-table [id=7996d607f2424ffeb69e7f49cf840299]) successfully deleted
I20260927 10:54:40.227100 2310 catalog_manager.cc:4841] TS 67984452b75a4ef7b56e0d0879d1958e (127.1.46.194:33407): tablet c7a69ec523ca4b58865b3003926652af (table test-table [id=7996d607f2424ffeb69e7f49cf840299]) successfully deleted
Sep 27 10:54:40 dist-test-slave-cffh krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506479, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:54:40.242287 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.230962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.1.46.193:45155)
0927 10:54:40.231088 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:40.231096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:40.231835 (+ 739us) server_negotiation.cc:407] Connection header received
0927 10:54:40.231971 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:40.231974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:40.232019 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:40.232104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:40.232774 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.233482 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.234283 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.234628 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.234732 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.238016 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:40.238027 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:40.238044 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:40.238067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:40.240216 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:40.240819 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:40.240825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:40.240827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:40.240932 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:40.241352 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:40.241356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:40.241358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:40.241634 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:40.241865 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:40.242016 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 10:54:40.242125 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 10:54:40.242796 2523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696
I20260927 10:54:40.252658 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:40.244610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.1.46.254:37467)
0927 10:54:40.244716 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:40.244720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:40.245011 (+ 291us) server_negotiation.cc:407] Connection header received
0927 10:54:40.245126 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:40.245128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:40.245169 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:40.245239 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:40.245935 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.246454 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.247592 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:40.247849 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:40.247932 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:40.249508 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:40.249511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:40.249513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:40.249535 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:40.251040 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:40.251461 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:40.251463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:40.251464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:40.251533 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:40.251828 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:40.251831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:40.251832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:40.252020 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:40.252131 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:40.252355 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 10:54:40.252450 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 10:54:40.253050 2325 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52056
I20260927 10:54:40.254361 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 2398
I20260927 10:54:40.261117 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 2538
I20260927 10:54:40.267889 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 2678
I20260927 10:54:40.272904 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 2294
2026-09-27T10:54:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2907 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:40 dist-test-slave-cffh krb5kdc[2891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:40 dist-test-slave-cffh krb5kdc[2891](info): set up 2 sockets
Sep 27 10:54:40 dist-test-slave-cffh krb5kdc[2891](info): commencing operation
krb5kdc: starting...
W20260927 10:54:41.298377 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:41 dist-test-slave-cffh krb5kdc[2891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:41.413312 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:38353
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:38353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:41.515280 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:41.515539 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:41.515609 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:41.519425 2904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:41.519497 2904 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:54:41.519529 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:41.519593 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:41.519630 2904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:41.519665 2904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:41.524508 2904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:38353
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:38353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:41.525429 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:41.526360 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:41.534195 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:41.534240 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:41.534250 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:41.534438 2904 server_base.cc:1034] running on GCE node
I20260927 10:54:41.534982 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:41.535926 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:41.537081 2904 hybrid_clock.cc:648] HybridClock initialized: now 1790506481537062 us; error 31 us; skew 500 ppm
Sep 27 10:54:41 dist-test-slave-cffh krb5kdc[2891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:41.539710 2904 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:41.540520 2904 webserver.cc:460] Webserver started at http://127.1.46.254:33977/ using document root <none> and password file <none>
I20260927 10:54:41.541010 2904 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:41.541096 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:41.541342 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:41.543280 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "06e4b63303ef47eb8099457a606cca1e"
format_stamp: "Formatted at 2026-09-27 10:54:41 on dist-test-slave-cffh"
server_key: "8ff6cded787e7cacac6c3070e28dc4f1"
server_key_iv: "d55b5b9f687d81596bc826b3fd9d4d72"
server_key_version: "encryptionkey@0"
I20260927 10:54:41.543818 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "06e4b63303ef47eb8099457a606cca1e"
format_stamp: "Formatted at 2026-09-27 10:54:41 on dist-test-slave-cffh"
server_key: "8ff6cded787e7cacac6c3070e28dc4f1"
server_key_iv: "d55b5b9f687d81596bc826b3fd9d4d72"
server_key_version: "encryptionkey@0"
I20260927 10:54:41.547212 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:41.549381 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:41.550529 2904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:41.550673 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "06e4b63303ef47eb8099457a606cca1e"
format_stamp: "Formatted at 2026-09-27 10:54:41 on dist-test-slave-cffh"
server_key: "8ff6cded787e7cacac6c3070e28dc4f1"
server_key_iv: "d55b5b9f687d81596bc826b3fd9d4d72"
server_key_version: "encryptionkey@0"
I20260927 10:54:41.550777 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:41.562157 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:41.564963 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:41.565155 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:41.572539 2904 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:38353
I20260927 10:54:41.572549 2981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:38353 every 8 connection(s)
I20260927 10:54:41.573374 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:41.576169 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:41.578066 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 2904
I20260927 10:54:41.578186 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:41.578447 1211 external_mini_cluster.cc:1428] Setting key a5dce7c75254568686461a5ac8a7eedb
I20260927 10:54:41.582283 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Bootstrap starting.
I20260927 10:54:41.584466 2982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:41.585197 2982 log.cc:824] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:54:41 dist-test-slave-cffh krb5kdc[2891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:41.587410 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: No bootstrap required, opened a new log
I20260927 10:54:41.590934 2982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4b63303ef47eb8099457a606cca1e" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38353 } }
I20260927 10:54:41.591190 2982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:41.591266 2982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e4b63303ef47eb8099457a606cca1e, State: Initialized, Role: FOLLOWER
I20260927 10:54:41.591763 2982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4b63303ef47eb8099457a606cca1e" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38353 } }
I20260927 10:54:41.591938 2982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:41.592007 2982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:41.592109 2982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:41.592952 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:41.579782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.1.46.254:38353)
0927 10:54:41.580179 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:41.580193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:41.580232 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:54:41.580822 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:41.580837 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:41.581147 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:41.581468 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:41.582243 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.583314 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.584184 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.584689 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.584846 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:41.587768 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:41.587773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:41.587784 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:41.587808 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:41.589928 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:41.590445 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:41.590449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:41.590452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:41.590561 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:41.590900 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:41.590903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:41.590904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:41.591369 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:41.591548 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:41.591825 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 10:54:41.592280 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260927 10:54:41.593004 2982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4b63303ef47eb8099457a606cca1e" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38353 } }
I20260927 10:54:41.593366 2982 leader_election.cc:302] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e4b63303ef47eb8099457a606cca1e; no voters:
I20260927 10:54:41.593714 2982 leader_election.cc:288] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:41.593811 2987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:41.594019 2987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [term 1 LEADER]: Becoming Leader. State: Replica: 06e4b63303ef47eb8099457a606cca1e, State: Running, Role: LEADER
I20260927 10:54:41.594296 2987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4b63303ef47eb8099457a606cca1e" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38353 } }
I20260927 10:54:41.594966 2982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:41.595515 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e4b63303ef47eb8099457a606cca1e. Latest consensus state: current_term: 1 leader_uuid: "06e4b63303ef47eb8099457a606cca1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4b63303ef47eb8099457a606cca1e" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38353 } } }
I20260927 10:54:41.595647 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:41.595842 2988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e4b63303ef47eb8099457a606cca1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4b63303ef47eb8099457a606cca1e" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38353 } } }
I20260927 10:54:41.595952 2988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:41.595985 2992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:41.598963 2992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:41.603706 2992 catalog_manager.cc:1347] Generated new cluster ID: 44f4a33a9a8449ac9fdaa5412bc13d9d
I20260927 10:54:41.603768 2992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:41.619495 2992 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:41.620599 2992 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:41.625020 2992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Generated new TSK 0
I20260927 10:54:41.625622 2992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:41.653070 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:38353
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:41.777558 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:41.777803 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:41.777846 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:41.781883 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:41.781936 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:41.781973 3008 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:54:41.781992 3008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:54:41.782023 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:41.787012 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:38353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:41.787856 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:41.788789 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:41.797844 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:41.797906 3008 server_base.cc:1034] running on GCE node
W20260927 10:54:41.797924 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:41.797832 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:41.798453 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:41.799068 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:41.800239 3008 hybrid_clock.cc:648] HybridClock initialized: now 1790506481800220 us; error 29 us; skew 500 ppm
Sep 27 10:54:41 dist-test-slave-cffh krb5kdc[2891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:41.803025 3008 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:41.803833 3008 webserver.cc:460] Webserver started at http://127.1.46.193:38349/ using document root <none> and password file <none>
I20260927 10:54:41.804358 3008 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:41.804415 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:41.804574 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:41.806417 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "6aaefa931d524ec1ac2156690015045d"
format_stamp: "Formatted at 2026-09-27 10:54:41 on dist-test-slave-cffh"
server_key: "d19da803a3e205bbdc648b953e26c504"
server_key_iv: "b87a49055d875d113a45f9b5322e6436"
server_key_version: "encryptionkey@0"
I20260927 10:54:41.807021 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "6aaefa931d524ec1ac2156690015045d"
format_stamp: "Formatted at 2026-09-27 10:54:41 on dist-test-slave-cffh"
server_key: "d19da803a3e205bbdc648b953e26c504"
server_key_iv: "b87a49055d875d113a45f9b5322e6436"
server_key_version: "encryptionkey@0"
I20260927 10:54:41.810251 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:41.812553 3023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:41.813683 3008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:41.813771 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "6aaefa931d524ec1ac2156690015045d"
format_stamp: "Formatted at 2026-09-27 10:54:41 on dist-test-slave-cffh"
server_key: "d19da803a3e205bbdc648b953e26c504"
server_key_iv: "b87a49055d875d113a45f9b5322e6436"
server_key_version: "encryptionkey@0"
I20260927 10:54:41.813843 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:41.832772 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:41.835659 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:41.835845 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:41.837173 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:41.837219 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:41.837255 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:41.837311 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:54:41 dist-test-slave-cffh krb5kdc[2891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:41.848315 3008 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:36263
I20260927 10:54:41.848392 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:36263 every 8 connection(s)
I20260927 10:54:41.849443 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:41.851478 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:41.838007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:40157 (local address 127.1.46.254:38353)
0927 10:54:41.838146 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:41.838151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:41.838947 (+ 796us) server_negotiation.cc:407] Connection header received
0927 10:54:41.839968 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:41.839974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:41.840048 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:41.840107 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:41.841523 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.842434 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.843706 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.844007 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.844152 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:41.847488 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:41.847492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:41.847494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:41.847518 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:41.849302 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:41.849936 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:41.849942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:41.849944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:41.850047 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:41.850425 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:41.850428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:41.850430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:41.850719 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:41.850799 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:41.851267 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 10:54:41.851364 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:54:41.852252 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:41.838312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:38353 (local address 127.1.46.193:40157)
0927 10:54:41.838804 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:41.838851 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:41.839728 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:41.840583 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:41.840592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:41.840905 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:41.841339 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:41.841351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.842618 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.842622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:41.843576 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:41.843584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.844195 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.844200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:41.844447 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:41.845147 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:41.845185 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:41.847291 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:41.849442 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:41.849446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:41.849470 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:41.849821 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:41.850184 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:41.850187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:41.850189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:41.850333 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:41.850830 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:41.850835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:41.851030 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 10:54:41.851170 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 10:54:41.851426 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
I20260927 10:54:41.853582 3139 heartbeater.cc:347] Connected to a master server at 127.1.46.254:38353
I20260927 10:54:41.853787 3139 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:41.854363 3139 heartbeater.cc:511] Master 127.1.46.254:38353 requested a full tablet report, sending...
I20260927 10:54:41.855937 2935 ts_manager.cc:194] Registered new tserver with Master: 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263)
I20260927 10:54:41.857404 2935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:40157
I20260927 10:54:41.859508 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 3008
I20260927 10:54:41.859643 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:41.859970 1211 external_mini_cluster.cc:1428] Setting key fbb7822989c82f91f64ea1bf140cef2e
I20260927 10:54:41.863582 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:41.855646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:38353 (local address 127.1.46.193:41915)
0927 10:54:41.855929 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:41.855946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:41.856054 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:41.856307 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:41.856310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:41.856516 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:41.856635 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:41.856642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.857401 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.857404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:41.858304 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:41.858312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.858899 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.858902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:41.858975 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:41.859446 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:41.859464 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:41.859928 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:41.861821 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:41.861824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:41.861826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:41.862116 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:41.862404 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:41.862411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:41.862416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:41.862531 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:41.862953 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:41.862960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:41.863076 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 10:54:41.863211 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 10:54:41.863387 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
I20260927 10:54:41.863701 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:41.855770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:41915 (local address 127.1.46.254:38353)
0927 10:54:41.855886 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:41.855889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:41.856005 (+ 116us) server_negotiation.cc:407] Connection header received
0927 10:54:41.856140 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:41.856144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:41.856185 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:41.856256 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:41.856766 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.857276 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.858411 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:41.858767 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:41.858889 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:41.860142 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:41.860147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:41.860151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:41.860179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:41.861709 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:41.862227 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:41.862229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:41.862231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:41.862307 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:41.862684 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:41.862687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:41.862688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:41.862851 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:41.862935 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:41.863308 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 10:54:41.863480 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:41.889179 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:38353
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:41.994602 3148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:41.994838 3148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:41.994925 3148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:41.999181 3148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:41.999228 3148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:41.999269 3148 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:54:41.999284 3148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:54:41.999312 3148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:42.004276 3148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:38353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:42.005175 3148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:42.006068 3148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:42.015101 3156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:42.015146 3154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:42.015101 3153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:42.015430 3148 server_base.cc:1034] running on GCE node
I20260927 10:54:42.015725 3148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:42.016285 3148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:42.017478 3148 hybrid_clock.cc:648] HybridClock initialized: now 1790506482017457 us; error 29 us; skew 500 ppm
Sep 27 10:54:42 dist-test-slave-cffh krb5kdc[2891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:42.020072 3148 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:42.020902 3148 webserver.cc:460] Webserver started at http://127.1.46.194:38977/ using document root <none> and password file <none>
I20260927 10:54:42.021447 3148 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:42.021538 3148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:42.021759 3148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:42.023757 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "b25cd8be2c1c4ffdb11fba5047349715"
format_stamp: "Formatted at 2026-09-27 10:54:42 on dist-test-slave-cffh"
server_key: "d1ca1f74a9ccd7063309a986a3d0ea7c"
server_key_iv: "7000299e2cb4d76cdc1304980f13933b"
server_key_version: "encryptionkey@0"
I20260927 10:54:42.024289 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "b25cd8be2c1c4ffdb11fba5047349715"
format_stamp: "Formatted at 2026-09-27 10:54:42 on dist-test-slave-cffh"
server_key: "d1ca1f74a9ccd7063309a986a3d0ea7c"
server_key_iv: "7000299e2cb4d76cdc1304980f13933b"
server_key_version: "encryptionkey@0"
I20260927 10:54:42.027695 3148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:42.029838 3163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:42.030961 3148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:42.031090 3148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "b25cd8be2c1c4ffdb11fba5047349715"
format_stamp: "Formatted at 2026-09-27 10:54:42 on dist-test-slave-cffh"
server_key: "d1ca1f74a9ccd7063309a986a3d0ea7c"
server_key_iv: "7000299e2cb4d76cdc1304980f13933b"
server_key_version: "encryptionkey@0"
I20260927 10:54:42.031198 3148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:42.041746 3148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:42.044826 3148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:42.045020 3148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:42.046344 3148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:42.046415 3148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:42.046485 3148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:42.046535 3148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:54:42 dist-test-slave-cffh krb5kdc[2891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:42.057014 3148 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:43515
I20260927 10:54:42.057057 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:43515 every 8 connection(s)
I20260927 10:54:42.057952 3148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:42.060536 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.047222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:52059 (local address 127.1.46.254:38353)
0927 10:54:42.047368 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.047373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.048227 (+ 854us) server_negotiation.cc:407] Connection header received
0927 10:54:42.049316 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.049321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.049379 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.049439 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:42.051149 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.052054 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.053756 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.054183 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.054349 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.057262 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:42.057265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:42.057267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:42.057290 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:42.058834 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.059267 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.059269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.059270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.059337 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.059632 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.059634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.059635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.059806 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:42.059878 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.060301 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.060407 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:54:42.061282 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.047614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:38353 (local address 127.1.46.194:52059)
0927 10:54:42.048059 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.048107 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.049063 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.049884 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.049895 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.050305 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:42.051000 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.051018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.052212 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.052218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.053622 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.053634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.054378 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.054382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.054605 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.055233 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:42.055258 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:42.057035 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:42.058945 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.058949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.058958 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.059180 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.059437 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.059439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.059441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.059556 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.059887 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:42.059892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:42.060076 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.060226 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.060460 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":134,"threads_started":1}
I20260927 10:54:42.062393 3279 heartbeater.cc:347] Connected to a master server at 127.1.46.254:38353
I20260927 10:54:42.062530 3279 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:42.063028 3279 heartbeater.cc:511] Master 127.1.46.254:38353 requested a full tablet report, sending...
I20260927 10:54:42.063966 2936 ts_manager.cc:194] Registered new tserver with Master: b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515)
I20260927 10:54:42.064457 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 3148
I20260927 10:54:42.064563 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:42.064721 2936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:52059
I20260927 10:54:42.064838 1211 external_mini_cluster.cc:1428] Setting key fbe0355e83e6fd2c192383ac89fac056
I20260927 10:54:42.071764 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.064352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:38353 (local address 127.1.46.194:52543)
0927 10:54:42.064626 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.064637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.064741 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.065055 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.065058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.065278 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:42.065396 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.065403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.066171 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.066175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.066771 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.066779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.067229 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.067233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.067329 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.067805 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:42.067816 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:42.068098 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:42.070174 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.070178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.070180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.070382 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.070714 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.070716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.070718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.070797 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.071333 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:42.071335 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:42.071410 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.071499 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.071596 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260927 10:54:42.072045 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.064387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:52543 (local address 127.1.46.254:38353)
0927 10:54:42.064490 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.064494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.064706 (+ 212us) server_negotiation.cc:407] Connection header received
0927 10:54:42.064826 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.064829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.064874 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.064955 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:42.065496 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.066061 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.066878 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.067120 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.067227 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.068254 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:42.068260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:42.068265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:42.068294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:42.070017 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.070486 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.070492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.070497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.070612 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.070900 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.070903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.070905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.071218 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:42.071336 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.071583 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.071704 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:42.093997 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:38353
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:42.201814 3288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:42.202031 3288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:42.202075 3288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:42.206007 3288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:42.206051 3288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:42.206089 3288 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:54:42.206101 3288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:54:42.206149 3288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:42.211148 3288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:38353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:42.212116 3288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:42.213037 3288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:42.222134 3294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:42.222102 3296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:42.222112 3293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:42.222344 3288 server_base.cc:1034] running on GCE node
I20260927 10:54:42.222860 3288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:42.223474 3288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:42.224726 3288 hybrid_clock.cc:648] HybridClock initialized: now 1790506482224701 us; error 40 us; skew 500 ppm
Sep 27 10:54:42 dist-test-slave-cffh krb5kdc[2891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:42.227402 3288 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:42.228169 3288 webserver.cc:460] Webserver started at http://127.1.46.195:38941/ using document root <none> and password file <none>
I20260927 10:54:42.228724 3288 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:42.228808 3288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:42.229022 3288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:42.231072 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "90a143649c3a479babf380f8e894640a"
format_stamp: "Formatted at 2026-09-27 10:54:42 on dist-test-slave-cffh"
server_key: "5795a31a07876d7f89dae4773f226319"
server_key_iv: "edfea5570ac1ace40228ceb848108219"
server_key_version: "encryptionkey@0"
I20260927 10:54:42.231590 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "90a143649c3a479babf380f8e894640a"
format_stamp: "Formatted at 2026-09-27 10:54:42 on dist-test-slave-cffh"
server_key: "5795a31a07876d7f89dae4773f226319"
server_key_iv: "edfea5570ac1ace40228ceb848108219"
server_key_version: "encryptionkey@0"
I20260927 10:54:42.235311 3288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:54:42.237655 3303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:42.238933 3288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:42.239055 3288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "90a143649c3a479babf380f8e894640a"
format_stamp: "Formatted at 2026-09-27 10:54:42 on dist-test-slave-cffh"
server_key: "5795a31a07876d7f89dae4773f226319"
server_key_iv: "edfea5570ac1ace40228ceb848108219"
server_key_version: "encryptionkey@0"
I20260927 10:54:42.239156 3288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:42.263664 3288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:42.266676 3288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:42.266875 3288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:42.268348 3288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:42.268425 3288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:42.268498 3288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:42.268549 3288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:54:42 dist-test-slave-cffh krb5kdc[2891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:42.278945 3288 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:44197
I20260927 10:54:42.279021 3418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:44197 every 8 connection(s)
I20260927 10:54:42.280097 3288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:42.282680 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.269179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:42229 (local address 127.1.46.254:38353)
0927 10:54:42.269332 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.269337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.270239 (+ 902us) server_negotiation.cc:407] Connection header received
0927 10:54:42.271452 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.271455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.271517 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.271605 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:42.273138 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.274077 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.275927 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.276232 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.276351 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.279093 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:42.279098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:42.279101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:42.279128 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:42.280751 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.281253 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.281256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.281257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.281335 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.281613 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.281616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.281617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.281820 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:42.281899 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.282392 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.282492 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:54:42.283448 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.269618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:38353 (local address 127.1.46.195:42229)
0927 10:54:42.270074 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.270122 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.271157 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.271794 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.271804 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.272306 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:42.272949 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.272968 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.274242 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.274247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.275762 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.275775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.276384 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.276389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.276557 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.277322 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:42.277355 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:42.278879 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:42.280887 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.280893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.280906 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.281145 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.281411 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.281413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.281414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.281525 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.281916 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:42.281921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:42.282125 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.282288 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.282514 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":131,"threads_started":1}
I20260927 10:54:42.284600 3419 heartbeater.cc:347] Connected to a master server at 127.1.46.254:38353
I20260927 10:54:42.284782 3419 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:42.285271 3419 heartbeater.cc:511] Master 127.1.46.254:38353 requested a full tablet report, sending...
I20260927 10:54:42.286310 2935 ts_manager.cc:194] Registered new tserver with Master: 90a143649c3a479babf380f8e894640a (127.1.46.195:44197)
I20260927 10:54:42.287101 2935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:42229
I20260927 10:54:42.290174 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 3288
I20260927 10:54:42.290302 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:42.290663 1211 external_mini_cluster.cc:1428] Setting key 7dbf89302dad4755a3f0ce5d15084933
I20260927 10:54:42.292795 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:54:42.294157 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.286864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:58399 (local address 127.1.46.254:38353)
0927 10:54:42.286967 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.286972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.287198 (+ 226us) server_negotiation.cc:407] Connection header received
0927 10:54:42.287308 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.287311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.287356 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.287461 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:42.287866 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.288363 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.289160 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.289395 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.289501 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.290697 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:42.290701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:42.290703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:42.290745 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:42.292461 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.293018 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.293020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.293021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.293093 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.293406 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.293408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.293409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.293551 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:42.293623 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.293961 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.294060 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 10:54:42.294390 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.286768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:38353 (local address 127.1.46.195:58399)
0927 10:54:42.287088 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.287099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.287196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.287488 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.287491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.287661 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:42.287755 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.287760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.288481 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.288485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.289045 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.289052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.289502 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.289505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.289596 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.290022 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:42.290037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:42.290428 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:42.292584 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.292590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.292595 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.292849 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.293197 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.293200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.293203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.293309 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.293648 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:42.293653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:42.293767 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.293901 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.294093 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":95,"threads_started":1}
Sep 27 10:54:42 dist-test-slave-cffh krb5kdc[2891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:54:42 dist-test-slave-cffh krb5kdc[2891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:42.318723 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.310791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.1.46.254:38353)
0927 10:54:42.310914 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.310917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.310930 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:54:42.310966 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.310968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.311003 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.311069 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:42.311783 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.312332 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.313126 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.313355 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.313433 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.315702 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:42.315706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:42.315708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:42.315729 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:42.317117 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.317605 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.317609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.317610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.317674 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.317911 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.317914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.317915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.318055 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:42.318153 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.318447 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.318527 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:54:42.321504 2935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:42.324482 2935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:42.338070 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.332159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:36263 (local address 127.0.0.1:51420)
0927 10:54:42.332495 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.332544 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.332652 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.333531 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.333539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.333561 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:42.333701 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.333709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.335067 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.335071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.336483 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.336493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.337524 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.337527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.337632 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.337688 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.337803 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.337925 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":25,"thread_start_us":85,"threads_started":1}
I20260927 10:54:42.338081 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.332298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.1.46.193:36263)
0927 10:54:42.332991 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.333012 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.333036 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:54:42.333117 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.333122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.333332 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.333544 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:42.333838 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.334977 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.336610 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.337410 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.337524 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.337683 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.337843 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.337934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":77,"threads_started":1}
I20260927 10:54:42.338407 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.332374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44197 (local address 127.0.0.1:36544)
0927 10:54:42.333318 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.333328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.333445 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.334244 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.334246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.334268 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:42.334343 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.334348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.335890 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.335894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.336993 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.337001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.338049 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.338054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.338141 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.338160 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.338204 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.338282 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":26,"thread_start_us":78,"threads_started":1}
I20260927 10:54:42.338690 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.332999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:43515 (local address 127.0.0.1:44452)
0927 10:54:42.333928 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.333976 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.334164 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.334819 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.334822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.334836 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:42.334910 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.334915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.336033 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.336036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.337166 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.337177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.338331 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.338334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.338409 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.338436 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.338506 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.338568 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":93,"threads_started":1}
I20260927 10:54:42.339010 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.332943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.1.46.195:44197)
0927 10:54:42.333224 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.333237 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.333416 (+ 179us) server_negotiation.cc:407] Connection header received
0927 10:54:42.333899 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.333905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.334090 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.334454 (+ 364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:42.334651 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.335734 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.337123 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.337898 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.338553 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.338726 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.338783 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.338883 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260927 10:54:42.339114 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.333378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.1.46.194:43515)
0927 10:54:42.333705 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.333716 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.334098 (+ 382us) server_negotiation.cc:407] Connection header received
0927 10:54:42.334312 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.334317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.334509 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.334681 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:42.335048 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.335922 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.337284 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.338049 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.338133 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.338277 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.338903 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.338981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":109,"threads_started":1}
I20260927 10:54:42.341325 3213 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8756e6caf1440db9669ff339ef744d (DEFAULT_TABLE table=test-table [id=5626b282f6904d71a2682e520ba89477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:42.341179 3073 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8756e6caf1440db9669ff339ef744d (DEFAULT_TABLE table=test-table [id=5626b282f6904d71a2682e520ba89477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:42.341521 3353 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8756e6caf1440db9669ff339ef744d (DEFAULT_TABLE table=test-table [id=5626b282f6904d71a2682e520ba89477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:42.342187 3213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8756e6caf1440db9669ff339ef744d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.342231 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8756e6caf1440db9669ff339ef744d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.342357 3353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8756e6caf1440db9669ff339ef744d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.348770 3442 tablet_bootstrap.cc:492] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Bootstrap starting.
I20260927 10:54:42.350056 3443 tablet_bootstrap.cc:492] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Bootstrap starting.
I20260927 10:54:42.350900 3444 tablet_bootstrap.cc:492] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Bootstrap starting.
I20260927 10:54:42.350951 3442 tablet_bootstrap.cc:654] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.351631 3442 log.cc:824] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:42.352092 3443 tablet_bootstrap.cc:654] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.352836 3444 tablet_bootstrap.cc:654] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.352901 3443 log.cc:824] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:42.353675 3442 tablet_bootstrap.cc:492] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: No bootstrap required, opened a new log
I20260927 10:54:42.353857 3442 ts_tablet_manager.cc:1397] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:54:42.354982 3443 tablet_bootstrap.cc:492] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: No bootstrap required, opened a new log
I20260927 10:54:42.355167 3443 ts_tablet_manager.cc:1397] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:54:42.356915 3442 raft_consensus.cc:348] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.357138 3442 raft_consensus.cc:374] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.357188 3442 raft_consensus.cc:729] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25cd8be2c1c4ffdb11fba5047349715, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.357641 3442 consensus_queue.cc:260] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.358457 3279 heartbeater.cc:503] Master 127.1.46.254:38353 was elected leader, sending a full tablet report...
I20260927 10:54:42.358642 3442 ts_tablet_manager.cc:1428] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:54:42.353720 3444 log.cc:824] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:42.359416 3443 raft_consensus.cc:348] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.359638 3443 raft_consensus.cc:374] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.359709 3443 raft_consensus.cc:729] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.360100 3443 consensus_queue.cc:260] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.361047 3443 ts_tablet_manager.cc:1428] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:54:42.361378 3139 heartbeater.cc:503] Master 127.1.46.254:38353 was elected leader, sending a full tablet report...
I20260927 10:54:42.362000 3444 tablet_bootstrap.cc:492] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: No bootstrap required, opened a new log
I20260927 10:54:42.362162 3444 ts_tablet_manager.cc:1397] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 10:54:42.365442 3444 raft_consensus.cc:348] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.365614 3444 raft_consensus.cc:374] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.365669 3444 raft_consensus.cc:729] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a143649c3a479babf380f8e894640a, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.366014 3444 consensus_queue.cc:260] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.366642 3419 heartbeater.cc:503] Master 127.1.46.254:38353 was elected leader, sending a full tablet report...
I20260927 10:54:42.366869 3444 ts_tablet_manager.cc:1428] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:54:42.531073 3420 tablet.cc:2369] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:54:42.559135 3280 tablet.cc:2369] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:54:42.574862 3450 raft_consensus.cc:482] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:42.575073 3450 raft_consensus.cc:504] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.576380 3450 leader_election.cc:288] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263), b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515)
I20260927 10:54:42.580735 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.576609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:36263 (local address 127.1.46.195:60163)
0927 10:54:42.576742 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.576757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.576850 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.577262 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.577266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.577288 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:42.577443 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.577452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.578518 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.578524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.579439 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.579448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.580342 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.580345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.580441 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.580464 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.580536 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.580602 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"spinlock_wait_cycles":1792}
I20260927 10:54:42.580806 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.576754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:60163 (local address 127.1.46.193:36263)
0927 10:54:42.576911 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.576916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.576934 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:54:42.576992 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.576995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.577047 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.577138 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:42.577588 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.578416 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.579570 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.580223 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.580359 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.580417 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.580569 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.580640 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 10:54:42.581473 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8756e6caf1440db9669ff339ef744d" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aaefa931d524ec1ac2156690015045d" is_pre_election: true
I20260927 10:54:42.581679 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.577089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:43259 (local address 127.1.46.194:43515)
0927 10:54:42.577200 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.577206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.577294 (+ 88us) server_negotiation.cc:407] Connection header received
0927 10:54:42.577455 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.577465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.577505 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.577595 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:42.578357 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.579173 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.580273 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.580854 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.580995 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.581057 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.581493 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.581564 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:54:42.581820 3451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.576893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:43515 (local address 127.1.46.195:43259)
0927 10:54:42.577199 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.577212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.577353 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.577690 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.577693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.577708 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:42.578215 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.578222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.579302 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.579305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.580144 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.580152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.581144 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.581148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.581241 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.581316 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.581474 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.581548 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"spinlock_wait_cycles":42624,"thread_start_us":97,"threads_started":1}
I20260927 10:54:42.581813 3093 raft_consensus.cc:2454] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 0.
I20260927 10:54:42.582263 3304 leader_election.cc:302] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, 90a143649c3a479babf380f8e894640a; no voters:
I20260927 10:54:42.582415 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8756e6caf1440db9669ff339ef744d" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715" is_pre_election: true
I20260927 10:54:42.582631 3450 raft_consensus.cc:2781] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:42.582729 3450 raft_consensus.cc:482] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:42.582720 3233 raft_consensus.cc:2454] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 0.
I20260927 10:54:42.582794 3450 raft_consensus.cc:3037] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:42.583886 3450 raft_consensus.cc:504] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.584244 3450 leader_election.cc:288] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 election: Requested vote from peers 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263), b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515)
I20260927 10:54:42.584656 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8756e6caf1440db9669ff339ef744d" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715"
I20260927 10:54:42.584667 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8756e6caf1440db9669ff339ef744d" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aaefa931d524ec1ac2156690015045d"
I20260927 10:54:42.584767 3233 raft_consensus.cc:3037] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:42.584798 3093 raft_consensus.cc:3037] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:42.585880 3093 raft_consensus.cc:2454] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 1.
I20260927 10:54:42.585955 3233 raft_consensus.cc:2454] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 1.
I20260927 10:54:42.586261 3304 leader_election.cc:302] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a143649c3a479babf380f8e894640a, b25cd8be2c1c4ffdb11fba5047349715; no voters:
I20260927 10:54:42.586539 3450 raft_consensus.cc:2781] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:42.586771 3450 raft_consensus.cc:686] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 1 LEADER]: Becoming Leader. State: Replica: 90a143649c3a479babf380f8e894640a, State: Running, Role: LEADER
I20260927 10:54:42.587091 3450 consensus_queue.cc:237] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } }
I20260927 10:54:42.590159 2936 catalog_manager.cc:5495] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a143649c3a479babf380f8e894640a (127.1.46.195). New cstate: current_term: 1 leader_uuid: "90a143649c3a479babf380f8e894640a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } health_report { overall_health: UNKNOWN } } }
W20260927 10:54:42.600834 3140 tablet.cc:2369] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:54:42 dist-test-slave-cffh krb5kdc[2891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:42.652344 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.644457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.1.46.254:38353)
0927 10:54:42.644601 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.644606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.644803 (+ 197us) server_negotiation.cc:407] Connection header received
0927 10:54:42.644874 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.644877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.644920 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.645006 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:42.645681 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.646211 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.647021 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.647239 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.647324 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.649565 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:42.649568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:42.649570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:42.649590 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:42.650853 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.651309 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.651312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.651313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.651386 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:42.651623 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:42.651625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:42.651626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:42.651792 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:42.651865 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.652127 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.652239 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":66}
I20260927 10:54:42.652351 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.644390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:38353 (local address 127.0.0.1:41254)
0927 10:54:42.644685 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.644698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.644804 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.645038 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.645040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.645451 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:42.645551 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.645557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.646316 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.646319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.646898 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.646907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.647341 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.647344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.647404 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.648009 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:42.648021 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:42.649443 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:42.650967 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.650975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.650980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.651211 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.651471 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:42.651473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:42.651474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:42.651547 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:42.651886 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:42.651891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:42.651982 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.652072 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.652180 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":132,"threads_started":1}
I20260927 10:54:42.656167 2936 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:41254:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:54:42.656867 2936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:42.661948 3073 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2ce9fa8d8f4c64a3773f0f946887a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5936fd63414f47eaaa80534614d351a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:54:42.661932 3213 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2ce9fa8d8f4c64a3773f0f946887a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5936fd63414f47eaaa80534614d351a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:54:42.661988 3353 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2ce9fa8d8f4c64a3773f0f946887a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5936fd63414f47eaaa80534614d351a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:54:42.662690 3213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2ce9fa8d8f4c64a3773f0f946887a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.662741 3353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2ce9fa8d8f4c64a3773f0f946887a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.662762 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2ce9fa8d8f4c64a3773f0f946887a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.665380 3443 tablet_bootstrap.cc:492] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: Bootstrap starting.
I20260927 10:54:42.665807 3444 tablet_bootstrap.cc:492] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: Bootstrap starting.
I20260927 10:54:42.666457 3443 tablet_bootstrap.cc:654] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.666953 3444 tablet_bootstrap.cc:654] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.667333 3442 tablet_bootstrap.cc:492] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: Bootstrap starting.
I20260927 10:54:42.667964 3443 tablet_bootstrap.cc:492] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: No bootstrap required, opened a new log
I20260927 10:54:42.668044 3443 ts_tablet_manager.cc:1397] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:42.668268 3442 tablet_bootstrap.cc:654] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.668273 3444 tablet_bootstrap.cc:492] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: No bootstrap required, opened a new log
I20260927 10:54:42.668344 3444 ts_tablet_manager.cc:1397] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:42.668515 3443 raft_consensus.cc:348] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.668635 3443 raft_consensus.cc:374] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.668669 3443 raft_consensus.cc:729] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.668800 3443 consensus_queue.cc:260] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.668841 3444 raft_consensus.cc:348] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.668934 3444 raft_consensus.cc:374] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.668972 3444 raft_consensus.cc:729] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a143649c3a479babf380f8e894640a, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.669111 3443 ts_tablet_manager.cc:1428] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:54:42.669103 3444 consensus_queue.cc:260] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.669359 3444 ts_tablet_manager.cc:1428] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 10:54:42.669499 3450 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.669636 3450 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:42.669759 3442 tablet_bootstrap.cc:492] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: No bootstrap required, opened a new log
I20260927 10:54:42.669844 3442 ts_tablet_manager.cc:1397] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:42.669852 3449 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.669994 3449 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:42.670346 3442 raft_consensus.cc:348] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.670459 3442 raft_consensus.cc:374] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.670496 3442 raft_consensus.cc:729] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25cd8be2c1c4ffdb11fba5047349715, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.670830 3442 consensus_queue.cc:260] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.671151 3442 ts_tablet_manager.cc:1428] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 10:54:42.671247 3447 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.671368 3447 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:42.716172 3449 raft_consensus.cc:482] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:42.716332 3449 raft_consensus.cc:504] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.717674 3449 leader_election.cc:288] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515), 90a143649c3a479babf380f8e894640a (127.1.46.195:44197)
I20260927 10:54:42.722071 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.718001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:38767 (local address 127.1.46.194:43515)
0927 10:54:42.718139 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.718144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.718352 (+ 208us) server_negotiation.cc:407] Connection header received
0927 10:54:42.718478 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.718481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.718537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.718643 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:42.718985 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.719724 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.720940 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.721426 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.721516 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.721555 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.721880 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.721945 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:54:42.722106 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.717871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:43515 (local address 127.1.46.193:38767)
0927 10:54:42.718253 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.718269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.718378 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.718672 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.718675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.718692 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:42.718851 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.718859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.719851 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.719855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.720815 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.720824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.721577 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.721581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.721720 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.721746 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.721841 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.721920 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260927 10:54:42.722400 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.718080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:47481 (local address 127.1.46.195:44197)
0927 10:54:42.718193 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.718198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.718478 (+ 280us) server_negotiation.cc:407] Connection header received
0927 10:54:42.718587 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.718589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.718669 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.718742 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:42.719484 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.720151 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.721248 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.721799 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.721917 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.721959 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.722150 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.722258 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:54:42.722679 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ce9fa8d8f4c64a3773f0f946887a1" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715" is_pre_election: true
I20260927 10:54:42.722829 3233 raft_consensus.cc:2454] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 0.
I20260927 10:54:42.723069 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.717949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44197 (local address 127.1.46.193:47481)
0927 10:54:42.718359 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.718374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.718481 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.718774 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.718778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.718794 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:42.719365 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.719375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.720276 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.720279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.721099 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.721107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.721914 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.721918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.722011 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.722032 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.722110 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.722366 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":113,"threads_started":1}
I20260927 10:54:42.723189 3024 leader_election.cc:302] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, b25cd8be2c1c4ffdb11fba5047349715; no voters:
I20260927 10:54:42.723476 3449 raft_consensus.cc:2781] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:42.723547 3449 raft_consensus.cc:482] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:42.723598 3449 raft_consensus.cc:3037] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:42.723691 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ce9fa8d8f4c64a3773f0f946887a1" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a143649c3a479babf380f8e894640a" is_pre_election: true
I20260927 10:54:42.723981 3373 raft_consensus.cc:2454] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 0.
I20260927 10:54:42.724819 3449 raft_consensus.cc:504] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.725215 3449 leader_election.cc:288] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 election: Requested vote from peers b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515), 90a143649c3a479babf380f8e894640a (127.1.46.195:44197)
I20260927 10:54:42.725533 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ce9fa8d8f4c64a3773f0f946887a1" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715"
I20260927 10:54:42.725551 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2ce9fa8d8f4c64a3773f0f946887a1" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a143649c3a479babf380f8e894640a"
I20260927 10:54:42.725651 3233 raft_consensus.cc:3037] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:42.725651 3373 raft_consensus.cc:3037] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:42.727257 3233 raft_consensus.cc:2454] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 1.
I20260927 10:54:42.727255 3373 raft_consensus.cc:2454] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 1.
I20260927 10:54:42.727596 3027 leader_election.cc:302] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, 90a143649c3a479babf380f8e894640a; no voters:
I20260927 10:54:42.727795 3449 raft_consensus.cc:2781] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:42.728041 3449 raft_consensus.cc:686] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [term 1 LEADER]: Becoming Leader. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Running, Role: LEADER
I20260927 10:54:42.728361 3449 consensus_queue.cc:237] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.730845 3465 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: TxnStatusTablet state changed. Reason: New leader 6aaefa931d524ec1ac2156690015045d. Latest consensus state: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.730954 3465 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:54:42.731274 3467 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:54:42.731534 3467 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:54:42.731614 2935 catalog_manager.cc:5495] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aaefa931d524ec1ac2156690015045d (127.1.46.193). New cstate: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:42.756052 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.751307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:36263 (local address 127.0.0.1:51908)
0927 10:54:42.751466 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.751484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.751584 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.751928 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.751931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.751936 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:54:42.752028 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.752033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.753154 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.753158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.754082 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.754093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.754793 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.754796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.754866 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.754962 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:54:42.755805 (+ 843us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:54:42.755817 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.755876 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.755941 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:54:42.756075 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.751438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.1.46.193:36263)
0927 10:54:42.751560 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.751564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.751579 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:54:42.751730 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.751733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.751808 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.751924 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:42.752140 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.752957 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.754246 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.754677 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.754780 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.755161 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:42.755703 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:42.755814 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.755890 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.755944 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:54:42.761597 3233 raft_consensus.cc:1261] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Refusing update from remote peer 6aaefa931d524ec1ac2156690015045d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:42.761596 3373 raft_consensus.cc:1261] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Refusing update from remote peer 6aaefa931d524ec1ac2156690015045d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:42.762290 3465 consensus_queue.cc:1035] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:42.762853 3449 consensus_queue.cc:1035] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:42.765710 3450 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: TxnStatusTablet state changed. Reason: New leader 6aaefa931d524ec1ac2156690015045d. Latest consensus state: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.765856 3450 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:42.767495 3447 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: TxnStatusTablet state changed. Reason: New leader 6aaefa931d524ec1ac2156690015045d. Latest consensus state: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.767622 3447 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:42.768263 3465 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.768361 3465 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:54:42.768594 3470 mvcc.cc:205] Tried to move back new op lower bound from 7333914553383837696 to 7333914553257861120. Current Snapshot: MvccSnapshot[applied={T|T < 7333914553383837696}]
I20260927 10:54:42.770314 3450 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.770428 3450 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 90a143649c3a479babf380f8e894640a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:42.770750 3471 mvcc.cc:205] Tried to move back new op lower bound from 7333914553383837696 to 7333914553257861120. Current Snapshot: MvccSnapshot[applied={T|T < 7333914553383837696}]
I20260927 10:54:42.772454 3449 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.772598 3449 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P 6aaefa931d524ec1ac2156690015045d: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:54:42.773655 3447 tablet_replica.cc:439] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } } }
I20260927 10:54:42.773763 3447 tablet_replica.cc:442] T 3e2ce9fa8d8f4c64a3773f0f946887a1 P b25cd8be2c1c4ffdb11fba5047349715: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:54:42.774115 3472 mvcc.cc:205] Tried to move back new op lower bound from 7333914553383837696 to 7333914553257861120. Current Snapshot: MvccSnapshot[applied={T|T < 7333914553383837696}]
I20260927 10:54:42.783944 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.780312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.1.46.195:44197)
0927 10:54:42.780434 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.780440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.780454 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:54:42.780624 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.780627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.780687 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.780822 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:42.781062 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.781576 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.782456 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.782776 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.782890 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.783152 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:42.783559 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:42.783684 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.783745 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.783806 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:54:42.794034 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.790708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:36263 (local address 127.1.46.195:57889)
0927 10:54:42.790865 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.790883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.790969 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.791233 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.791235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.791241 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:54:42.791328 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.791332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.792286 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.792289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.792912 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.792920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.793457 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.793459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.793522 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.793557 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:54:42.793834 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:54:42.793839 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.793886 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.793939 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:54:42.794064 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.790824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:57889 (local address 127.1.46.193:36263)
0927 10:54:42.790948 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.790953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.790967 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:54:42.791094 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.791097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.791142 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.791218 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:42.791420 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.792171 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.793016 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.793354 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.793451 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.793684 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:42.793756 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:42.793823 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.793899 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.793943 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:54:42.801141 3233 raft_consensus.cc:1261] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Refusing update from remote peer 90a143649c3a479babf380f8e894640a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:42.801324 3093 raft_consensus.cc:1261] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Refusing update from remote peer 90a143649c3a479babf380f8e894640a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:42.801631 3452 consensus_queue.cc:1035] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:42.801848 3450 consensus_queue.cc:1035] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:42.828974 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.824529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:48171 (local address 127.1.46.195:44197)
0927 10:54:42.824667 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:42.824673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:42.824893 (+ 220us) server_negotiation.cc:407] Connection header received
0927 10:54:42.825050 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:42.825054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:42.825101 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:42.825144 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:42.825549 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.826272 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.827552 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.827780 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.827867 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.828177 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:42.828273 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:42.828371 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:42.828745 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 10:54:42.828796 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:54:42.828991 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:42.824427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44197 (local address 127.1.46.193:48171)
0927 10:54:42.824792 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:42.824807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:42.824933 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:42.825320 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:42.825323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:42.825331 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:54:42.825449 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.825455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.826402 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.826406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.827423 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:42.827434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:42.827882 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:42.827886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:42.827981 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:42.828031 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:54:42.828616 (+ 585us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:54:42.828622 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:54:42.828719 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:54:42.828816 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":137,"threads_started":1}
I20260927 10:54:42.855722 2934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:42.855931 2934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:42.858525 2934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Sending DeleteTablet for 3 replicas of tablet 0b8756e6caf1440db9669ff339ef744d
I20260927 10:54:42.859293 3073 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8756e6caf1440db9669ff339ef744d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:42 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:51420
I20260927 10:54:42.859699 3213 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8756e6caf1440db9669ff339ef744d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:42 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44452
I20260927 10:54:42.859951 3488 tablet_replica.cc:330] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: stopping tablet replica
I20260927 10:54:42.860309 3488 raft_consensus.cc:2229] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:42.860555 3488 raft_consensus.cc:2258] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:42.861016 3353 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8756e6caf1440db9669ff339ef744d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:42 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:36544
I20260927 10:54:42.861505 2934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:42.862004 2934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:42.862476 3488 ts_tablet_manager.cc:1905] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:42.865870 3488 ts_tablet_manager.cc:1918] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:42.866195 3489 tablet_replica.cc:330] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: stopping tablet replica
I20260927 10:54:42.866611 3489 raft_consensus.cc:2229] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:42.866900 3489 raft_consensus.cc:2258] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:42.868177 3213 tablet_service.cc:1467] Processing CreateTablet for tablet 144f62d3847840b4b6e00c523346a1b4 (DEFAULT_TABLE table=test-table [id=dc7efaff8e294e1a9a50d1d093493346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:42.868059 3073 tablet_service.cc:1467] Processing CreateTablet for tablet 144f62d3847840b4b6e00c523346a1b4 (DEFAULT_TABLE table=test-table [id=dc7efaff8e294e1a9a50d1d093493346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:42.868489 3213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144f62d3847840b4b6e00c523346a1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.868507 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144f62d3847840b4b6e00c523346a1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.870891 3443 tablet_bootstrap.cc:492] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: Bootstrap starting.
I20260927 10:54:42.870983 3442 tablet_bootstrap.cc:492] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: Bootstrap starting.
I20260927 10:54:42.871578 3353 tablet_service.cc:1467] Processing CreateTablet for tablet 144f62d3847840b4b6e00c523346a1b4 (DEFAULT_TABLE table=test-table [id=dc7efaff8e294e1a9a50d1d093493346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:42.871881 3443 tablet_bootstrap.cc:654] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.871892 3353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144f62d3847840b4b6e00c523346a1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:42.872032 3442 tablet_bootstrap.cc:654] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.873047 3488 log.cc:1196] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/0b8756e6caf1440db9669ff339ef744d
I20260927 10:54:42.873665 3489 ts_tablet_manager.cc:1905] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:42.874127 3488 ts_tablet_manager.cc:1939] T 0b8756e6caf1440db9669ff339ef744d P 6aaefa931d524ec1ac2156690015045d: Deleting consensus metadata
I20260927 10:54:42.874497 3444 tablet_bootstrap.cc:492] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: Bootstrap starting.
I20260927 10:54:42.875566 3444 tablet_bootstrap.cc:654] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:42.876648 2920 catalog_manager.cc:4841] TS 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263): tablet 0b8756e6caf1440db9669ff339ef744d (table test-table [id=5626b282f6904d71a2682e520ba89477]) successfully deleted
I20260927 10:54:42.879488 3489 ts_tablet_manager.cc:1918] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:42.880414 3442 tablet_bootstrap.cc:492] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: No bootstrap required, opened a new log
I20260927 10:54:42.880538 3442 ts_tablet_manager.cc:1397] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 10:54:42.880981 3442 raft_consensus.cc:348] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.881085 3442 raft_consensus.cc:374] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.881116 3442 raft_consensus.cc:729] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25cd8be2c1c4ffdb11fba5047349715, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.881248 3442 consensus_queue.cc:260] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.881513 3442 ts_tablet_manager.cc:1428] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:42.883679 3489 log.cc:1196] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/0b8756e6caf1440db9669ff339ef744d
I20260927 10:54:42.884095 3489 ts_tablet_manager.cc:1939] T 0b8756e6caf1440db9669ff339ef744d P b25cd8be2c1c4ffdb11fba5047349715: Deleting consensus metadata
I20260927 10:54:42.884708 3444 tablet_bootstrap.cc:492] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: No bootstrap required, opened a new log
I20260927 10:54:42.884815 3444 ts_tablet_manager.cc:1397] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 10:54:42.885331 3444 raft_consensus.cc:348] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.885466 3444 raft_consensus.cc:374] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.885512 3444 raft_consensus.cc:729] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a143649c3a479babf380f8e894640a, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.885555 3443 tablet_bootstrap.cc:492] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: No bootstrap required, opened a new log
I20260927 10:54:42.885646 3443 ts_tablet_manager.cc:1397] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260927 10:54:42.885663 3444 consensus_queue.cc:260] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.886047 3444 ts_tablet_manager.cc:1428] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:42.886112 3443 raft_consensus.cc:348] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.886243 3443 raft_consensus.cc:374] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:42.886288 3443 raft_consensus.cc:729] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Initialized, Role: FOLLOWER
I20260927 10:54:42.886417 3443 consensus_queue.cc:260] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:42.886701 3443 ts_tablet_manager.cc:1428] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:54:42.886823 2920 catalog_manager.cc:4841] TS b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515): tablet 0b8756e6caf1440db9669ff339ef744d (table test-table [id=5626b282f6904d71a2682e520ba89477]) successfully deleted
I20260927 10:54:42.891526 3490 tablet_replica.cc:330] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: stopping tablet replica
I20260927 10:54:42.891882 3490 raft_consensus.cc:2229] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:42.892256 3490 raft_consensus.cc:2258] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:42.893563 3490 ts_tablet_manager.cc:1905] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:42.897468 3490 ts_tablet_manager.cc:1918] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:42.897598 3490 log.cc:1196] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/0b8756e6caf1440db9669ff339ef744d
I20260927 10:54:42.897919 3490 ts_tablet_manager.cc:1939] T 0b8756e6caf1440db9669ff339ef744d P 90a143649c3a479babf380f8e894640a: Deleting consensus metadata
I20260927 10:54:42.899024 2923 catalog_manager.cc:4841] TS 90a143649c3a479babf380f8e894640a (127.1.46.195:44197): tablet 0b8756e6caf1440db9669ff339ef744d (table test-table [id=5626b282f6904d71a2682e520ba89477]) successfully deleted
I20260927 10:54:43.059903 3465 raft_consensus.cc:482] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:43.060035 3465 raft_consensus.cc:504] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:43.060541 3465 leader_election.cc:288] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515), 90a143649c3a479babf380f8e894640a (127.1.46.195:44197)
I20260927 10:54:43.060974 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144f62d3847840b4b6e00c523346a1b4" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715" is_pre_election: true
I20260927 10:54:43.061015 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144f62d3847840b4b6e00c523346a1b4" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a143649c3a479babf380f8e894640a" is_pre_election: true
I20260927 10:54:43.061112 3233 raft_consensus.cc:2454] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 0.
I20260927 10:54:43.061177 3373 raft_consensus.cc:2454] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 0.
I20260927 10:54:43.061395 3024 leader_election.cc:302] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, b25cd8be2c1c4ffdb11fba5047349715; no voters:
I20260927 10:54:43.061589 3465 raft_consensus.cc:2781] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:43.061652 3465 raft_consensus.cc:482] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:43.061687 3465 raft_consensus.cc:3037] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:43.062978 3465 raft_consensus.cc:504] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:43.063319 3465 leader_election.cc:288] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 election: Requested vote from peers b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515), 90a143649c3a479babf380f8e894640a (127.1.46.195:44197)
I20260927 10:54:43.063634 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144f62d3847840b4b6e00c523346a1b4" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715"
I20260927 10:54:43.063771 3233 raft_consensus.cc:3037] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:43.063738 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144f62d3847840b4b6e00c523346a1b4" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a143649c3a479babf380f8e894640a"
I20260927 10:54:43.063885 3373 raft_consensus.cc:3037] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:43.064788 3373 raft_consensus.cc:2454] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 1.
I20260927 10:54:43.064841 3233 raft_consensus.cc:2454] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 1.
I20260927 10:54:43.065068 3027 leader_election.cc:302] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, 90a143649c3a479babf380f8e894640a; no voters:
I20260927 10:54:43.065235 3465 raft_consensus.cc:2781] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:43.065332 3465 raft_consensus.cc:686] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 1 LEADER]: Becoming Leader. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Running, Role: LEADER
I20260927 10:54:43.065455 3465 consensus_queue.cc:237] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:43.067087 2935 catalog_manager.cc:5495] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aaefa931d524ec1ac2156690015045d (127.1.46.193). New cstate: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:43.126965 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:43.123875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.1.46.193:36263)
0927 10:54:43.124000 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:43.124004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:43.124043 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:54:43.124171 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:43.124173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:43.124211 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:43.124265 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:43.124478 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:43.125034 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:43.125902 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:43.126101 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:43.126173 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:43.126426 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:43.126487 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:43.126538 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:43.126757 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 10:54:43.126797 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:54:43.135272 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:43.132571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:36263 (local address 127.1.46.193:42923)
0927 10:54:43.132729 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:43.132743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:43.132825 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:43.133009 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:43.133010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:43.133015 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:54:43.133085 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:43.133088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:43.133676 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:43.133678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:43.134231 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:43.134239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:43.134693 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:43.134695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:43.134761 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:43.134783 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:54:43.135027 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:54:43.135032 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:54:43.135081 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 10:54:43.135143 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":1}
I20260927 10:54:43.135293 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:43.132545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:42923 (local address 127.1.46.193:36263)
0927 10:54:43.132659 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:43.132662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:43.132801 (+ 139us) server_negotiation.cc:407] Connection header received
0927 10:54:43.132885 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:43.132887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:43.132925 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:43.132992 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:43.133145 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:43.133589 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:43.134329 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:43.134521 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:43.134710 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:43.134881 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:43.134939 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:43.134998 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:43.135097 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 10:54:43.135139 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:54:43.140399 3233 raft_consensus.cc:1261] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Refusing update from remote peer 6aaefa931d524ec1ac2156690015045d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:43.140525 3373 raft_consensus.cc:1261] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Refusing update from remote peer 6aaefa931d524ec1ac2156690015045d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:43.140941 3449 consensus_queue.cc:1035] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:43.141131 3465 consensus_queue.cc:1035] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:54:43.156365 3470 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:54:44.710295 2936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:44.710456 2936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:44.712877 2936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Sending DeleteTablet for 3 replicas of tablet 144f62d3847840b4b6e00c523346a1b4
I20260927 10:54:44.713446 3072 tablet_service.cc:1514] Processing DeleteTablet for tablet 144f62d3847840b4b6e00c523346a1b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:44 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:51420
I20260927 10:54:44.713524 3353 tablet_service.cc:1514] Processing DeleteTablet for tablet 144f62d3847840b4b6e00c523346a1b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:44 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:36544
I20260927 10:54:44.713598 3213 tablet_service.cc:1514] Processing DeleteTablet for tablet 144f62d3847840b4b6e00c523346a1b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:44 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44452
I20260927 10:54:44.713781 3519 tablet_replica.cc:330] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: stopping tablet replica
I20260927 10:54:44.713917 3519 raft_consensus.cc:2229] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:44.714134 3521 tablet_replica.cc:330] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: stopping tablet replica
I20260927 10:54:44.714249 3519 raft_consensus.cc:2258] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:44.714427 3521 raft_consensus.cc:2229] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:44.714617 3521 raft_consensus.cc:2258] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:44.714951 3519 ts_tablet_manager.cc:1905] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:44.715341 3521 ts_tablet_manager.cc:1905] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:44.717768 3519 ts_tablet_manager.cc:1918] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:54:44.717854 3519 log.cc:1196] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/144f62d3847840b4b6e00c523346a1b4
I20260927 10:54:44.717847 3521 ts_tablet_manager.cc:1918] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:54:44.717907 3521 log.cc:1196] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/144f62d3847840b4b6e00c523346a1b4
I20260927 10:54:44.718169 3519 ts_tablet_manager.cc:1939] T 144f62d3847840b4b6e00c523346a1b4 P 6aaefa931d524ec1ac2156690015045d: Deleting consensus metadata
I20260927 10:54:44.718185 3521 ts_tablet_manager.cc:1939] T 144f62d3847840b4b6e00c523346a1b4 P b25cd8be2c1c4ffdb11fba5047349715: Deleting consensus metadata
I20260927 10:54:44.718766 3520 tablet_replica.cc:330] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: stopping tablet replica
I20260927 10:54:44.718853 2920 catalog_manager.cc:4841] TS 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263): tablet 144f62d3847840b4b6e00c523346a1b4 (table test-table [id=dc7efaff8e294e1a9a50d1d093493346]) successfully deleted
I20260927 10:54:44.718936 3520 raft_consensus.cc:2229] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:44.718994 2920 catalog_manager.cc:4841] TS b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515): tablet 144f62d3847840b4b6e00c523346a1b4 (table test-table [id=dc7efaff8e294e1a9a50d1d093493346]) successfully deleted
I20260927 10:54:44.719054 3520 raft_consensus.cc:2258] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:44.719624 3520 ts_tablet_manager.cc:1905] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:44.721845 3520 ts_tablet_manager.cc:1918] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:54:44.721935 3520 log.cc:1196] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/144f62d3847840b4b6e00c523346a1b4
I20260927 10:54:44.722256 3520 ts_tablet_manager.cc:1939] T 144f62d3847840b4b6e00c523346a1b4 P 90a143649c3a479babf380f8e894640a: Deleting consensus metadata
I20260927 10:54:44.723045 2923 catalog_manager.cc:4841] TS 90a143649c3a479babf380f8e894640a (127.1.46.195:44197): tablet 144f62d3847840b4b6e00c523346a1b4 (table test-table [id=dc7efaff8e294e1a9a50d1d093493346]) successfully deleted
Sep 27 10:54:44 dist-test-slave-cffh krb5kdc[2891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:54:44 dist-test-slave-cffh krb5kdc[2891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:44.749356 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:44.733915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.1.46.254:38353)
0927 10:54:44.734236 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:44.734243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:44.734258 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:54:44.734397 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:44.734399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:44.734449 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:44.734529 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:44.735240 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:44.735807 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:44.736645 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:44.736889 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:44.736970 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:44.746196 (+ 9226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:44.746199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:44.746202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:44.746233 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:44.747696 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:44.748097 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:44.748099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:44.748100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:44.748172 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:44.748413 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:44.748415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:44.748416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:44.748601 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:44.748702 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:44.748963 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 10:54:44.749142 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":118,"threads_started":1}
I20260927 10:54:44.752161 2936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:44.752635 2936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:44.757608 3072 tablet_service.cc:1467] Processing CreateTablet for tablet c25505e378c5402e934820fd356fd2d6 (DEFAULT_TABLE table=test-table [id=905f95f9ab8f43ed9bde3b79d3bcd339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:44.757678 3213 tablet_service.cc:1467] Processing CreateTablet for tablet c25505e378c5402e934820fd356fd2d6 (DEFAULT_TABLE table=test-table [id=905f95f9ab8f43ed9bde3b79d3bcd339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:44.757964 3072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25505e378c5402e934820fd356fd2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:44.757966 3213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25505e378c5402e934820fd356fd2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:44.760097 3530 tablet_bootstrap.cc:492] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: Bootstrap starting.
I20260927 10:54:44.760330 3353 tablet_service.cc:1467] Processing CreateTablet for tablet c25505e378c5402e934820fd356fd2d6 (DEFAULT_TABLE table=test-table [id=905f95f9ab8f43ed9bde3b79d3bcd339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:44.760608 3353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25505e378c5402e934820fd356fd2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:44.761350 3530 tablet_bootstrap.cc:654] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:44.762708 3530 tablet_bootstrap.cc:492] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: No bootstrap required, opened a new log
I20260927 10:54:44.762809 3530 ts_tablet_manager.cc:1397] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:44.763018 3531 tablet_bootstrap.cc:492] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: Bootstrap starting.
I20260927 10:54:44.763274 3530 raft_consensus.cc:348] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.763392 3530 raft_consensus.cc:374] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:44.763432 3530 raft_consensus.cc:729] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25cd8be2c1c4ffdb11fba5047349715, State: Initialized, Role: FOLLOWER
I20260927 10:54:44.763595 3530 consensus_queue.cc:260] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.763989 3530 ts_tablet_manager.cc:1428] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:44.764175 3531 tablet_bootstrap.cc:654] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:44.764350 3533 tablet_bootstrap.cc:492] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: Bootstrap starting.
I20260927 10:54:44.765196 3533 tablet_bootstrap.cc:654] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:44.766036 3531 tablet_bootstrap.cc:492] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: No bootstrap required, opened a new log
I20260927 10:54:44.766156 3531 ts_tablet_manager.cc:1397] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:44.766355 3533 tablet_bootstrap.cc:492] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: No bootstrap required, opened a new log
I20260927 10:54:44.766422 3533 ts_tablet_manager.cc:1397] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:44.766661 3531 raft_consensus.cc:348] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.766785 3531 raft_consensus.cc:374] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:44.766829 3531 raft_consensus.cc:729] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Initialized, Role: FOLLOWER
I20260927 10:54:44.766975 3533 raft_consensus.cc:348] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.767012 3531 consensus_queue.cc:260] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.767083 3533 raft_consensus.cc:374] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:44.767335 3531 ts_tablet_manager.cc:1428] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:44.767123 3533 raft_consensus.cc:729] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a143649c3a479babf380f8e894640a, State: Initialized, Role: FOLLOWER
I20260927 10:54:44.767732 3533 consensus_queue.cc:260] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.768088 3533 ts_tablet_manager.cc:1428] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:44.991930 3514 raft_consensus.cc:482] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:44.992075 3514 raft_consensus.cc:504] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.992585 3514 leader_election.cc:288] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515), 90a143649c3a479babf380f8e894640a (127.1.46.195:44197)
I20260927 10:54:44.993043 3232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25505e378c5402e934820fd356fd2d6" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715" is_pre_election: true
I20260927 10:54:44.993145 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25505e378c5402e934820fd356fd2d6" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a143649c3a479babf380f8e894640a" is_pre_election: true
I20260927 10:54:44.993203 3232 raft_consensus.cc:2454] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 0.
I20260927 10:54:44.993276 3373 raft_consensus.cc:2454] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 0.
I20260927 10:54:44.993595 3024 leader_election.cc:302] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, 90a143649c3a479babf380f8e894640a, b25cd8be2c1c4ffdb11fba5047349715; no voters:
I20260927 10:54:44.993790 3514 raft_consensus.cc:2781] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:44.993851 3514 raft_consensus.cc:482] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:44.993878 3514 raft_consensus.cc:3037] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:44.995004 3514 raft_consensus.cc:504] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.995333 3514 leader_election.cc:288] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 election: Requested vote from peers b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515), 90a143649c3a479babf380f8e894640a (127.1.46.195:44197)
I20260927 10:54:44.995673 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25505e378c5402e934820fd356fd2d6" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a143649c3a479babf380f8e894640a"
I20260927 10:54:44.995673 3232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25505e378c5402e934820fd356fd2d6" candidate_uuid: "6aaefa931d524ec1ac2156690015045d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715"
I20260927 10:54:44.995792 3232 raft_consensus.cc:3037] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:44.995800 3373 raft_consensus.cc:3037] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:44.996918 3373 raft_consensus.cc:2454] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 1.
I20260927 10:54:44.996918 3232 raft_consensus.cc:2454] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaefa931d524ec1ac2156690015045d in term 1.
I20260927 10:54:44.997196 3027 leader_election.cc:302] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, 90a143649c3a479babf380f8e894640a; no voters:
I20260927 10:54:44.997359 3514 raft_consensus.cc:2781] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:44.997470 3514 raft_consensus.cc:686] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 1 LEADER]: Becoming Leader. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Running, Role: LEADER
I20260927 10:54:44.997614 3514 consensus_queue.cc:237] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:44.999248 2935 catalog_manager.cc:5495] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aaefa931d524ec1ac2156690015045d (127.1.46.193). New cstate: current_term: 1 leader_uuid: "6aaefa931d524ec1ac2156690015045d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:45.063978 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:45.059952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.1.46.193:36263)
0927 10:54:45.060330 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:45.060338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:45.060350 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:54:45.060392 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:45.060395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:45.060443 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:45.060523 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:45.060815 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:45.061623 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:45.062613 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:45.062966 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:45.063290 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:45.063560 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:45.063639 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:45.063709 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:45.063793 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 10:54:45.063842 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":84,"threads_started":1}
I20260927 10:54:45.069926 3373 raft_consensus.cc:1261] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Refusing update from remote peer 6aaefa931d524ec1ac2156690015045d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:45.070077 3232 raft_consensus.cc:1261] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Refusing update from remote peer 6aaefa931d524ec1ac2156690015045d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:45.070389 3501 consensus_queue.cc:1035] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:45.070604 3477 consensus_queue.cc:1035] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:45.102049 2936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:45.102205 2936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:45.104215 2936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Sending DeleteTablet for 3 replicas of tablet c25505e378c5402e934820fd356fd2d6
I20260927 10:54:45.104880 3072 tablet_service.cc:1514] Processing DeleteTablet for tablet c25505e378c5402e934820fd356fd2d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:45 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:51420
I20260927 10:54:45.104879 3213 tablet_service.cc:1514] Processing DeleteTablet for tablet c25505e378c5402e934820fd356fd2d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:45 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44452
I20260927 10:54:45.104933 3353 tablet_service.cc:1514] Processing DeleteTablet for tablet c25505e378c5402e934820fd356fd2d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:45 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:36544
I20260927 10:54:45.105113 3519 tablet_replica.cc:330] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: stopping tablet replica
I20260927 10:54:45.105149 3520 tablet_replica.cc:330] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: stopping tablet replica
I20260927 10:54:45.105149 3521 tablet_replica.cc:330] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: stopping tablet replica
I20260927 10:54:45.105226 3519 raft_consensus.cc:2229] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:45.105254 3520 raft_consensus.cc:2229] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:45.105262 3521 raft_consensus.cc:2229] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:45.105355 3520 raft_consensus.cc:2258] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:45.105376 3521 raft_consensus.cc:2258] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:45.105418 3519 raft_consensus.cc:2258] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:45.106220 3521 ts_tablet_manager.cc:1905] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:45.106264 2936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:54:45.106410 3520 ts_tablet_manager.cc:1905] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:54:45.106742 2936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:45.107869 3519 ts_tablet_manager.cc:1905] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:45.111822 3072 tablet_service.cc:1467] Processing CreateTablet for tablet 9939721e696b43fd8d2b9cd809a1fab2 (DEFAULT_TABLE table=test-table [id=a9d085555f714d30a7dc4ef669d4cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:45.111900 3213 tablet_service.cc:1467] Processing CreateTablet for tablet 9939721e696b43fd8d2b9cd809a1fab2 (DEFAULT_TABLE table=test-table [id=a9d085555f714d30a7dc4ef669d4cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:45.111900 3353 tablet_service.cc:1467] Processing CreateTablet for tablet 9939721e696b43fd8d2b9cd809a1fab2 (DEFAULT_TABLE table=test-table [id=a9d085555f714d30a7dc4ef669d4cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:45.113018 3213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9939721e696b43fd8d2b9cd809a1fab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:45.113014 3521 ts_tablet_manager.cc:1918] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:45.113206 3521 log.cc:1196] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/c25505e378c5402e934820fd356fd2d6
I20260927 10:54:45.113296 3353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9939721e696b43fd8d2b9cd809a1fab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:45.113263 3519 ts_tablet_manager.cc:1918] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:45.113358 3519 log.cc:1196] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/c25505e378c5402e934820fd356fd2d6
I20260927 10:54:45.113507 3521 ts_tablet_manager.cc:1939] T c25505e378c5402e934820fd356fd2d6 P b25cd8be2c1c4ffdb11fba5047349715: Deleting consensus metadata
I20260927 10:54:45.113615 3519 ts_tablet_manager.cc:1939] T c25505e378c5402e934820fd356fd2d6 P 6aaefa931d524ec1ac2156690015045d: Deleting consensus metadata
I20260927 10:54:45.113806 3072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9939721e696b43fd8d2b9cd809a1fab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:45.114480 2920 catalog_manager.cc:4841] TS b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515): tablet c25505e378c5402e934820fd356fd2d6 (table test-table [id=905f95f9ab8f43ed9bde3b79d3bcd339]) successfully deleted
I20260927 10:54:45.115401 3533 tablet_bootstrap.cc:492] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: Bootstrap starting.
I20260927 10:54:45.115573 3530 tablet_bootstrap.cc:492] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715: Bootstrap starting.
I20260927 10:54:45.115877 3520 ts_tablet_manager.cc:1918] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:54:45.115990 3520 log.cc:1196] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/c25505e378c5402e934820fd356fd2d6
I20260927 10:54:45.116256 3533 tablet_bootstrap.cc:654] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:45.116348 3520 ts_tablet_manager.cc:1939] T c25505e378c5402e934820fd356fd2d6 P 90a143649c3a479babf380f8e894640a: Deleting consensus metadata
I20260927 10:54:45.116559 3530 tablet_bootstrap.cc:654] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:45.116667 3531 tablet_bootstrap.cc:492] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d: Bootstrap starting.
I20260927 10:54:45.117148 2923 catalog_manager.cc:4841] TS 90a143649c3a479babf380f8e894640a (127.1.46.195:44197): tablet c25505e378c5402e934820fd356fd2d6 (table test-table [id=905f95f9ab8f43ed9bde3b79d3bcd339]) successfully deleted
I20260927 10:54:45.117389 3533 tablet_bootstrap.cc:492] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: No bootstrap required, opened a new log
I20260927 10:54:45.117472 3533 ts_tablet_manager.cc:1397] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:45.117504 3531 tablet_bootstrap.cc:654] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:45.117658 3530 tablet_bootstrap.cc:492] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715: No bootstrap required, opened a new log
I20260927 10:54:45.117745 3530 ts_tablet_manager.cc:1397] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:45.117937 3533 raft_consensus.cc:348] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.118073 3533 raft_consensus.cc:374] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:45.118130 3533 raft_consensus.cc:729] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a143649c3a479babf380f8e894640a, State: Initialized, Role: FOLLOWER
I20260927 10:54:45.118223 2920 catalog_manager.cc:4841] TS 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263): tablet c25505e378c5402e934820fd356fd2d6 (table test-table [id=905f95f9ab8f43ed9bde3b79d3bcd339]) successfully deleted
I20260927 10:54:45.118201 3530 raft_consensus.cc:348] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.118312 3530 raft_consensus.cc:374] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:45.118269 3533 consensus_queue.cc:260] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.118353 3530 raft_consensus.cc:729] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25cd8be2c1c4ffdb11fba5047349715, State: Initialized, Role: FOLLOWER
I20260927 10:54:45.118470 3530 consensus_queue.cc:260] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.118620 3533 ts_tablet_manager.cc:1428] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 10:54:45.118752 3530 ts_tablet_manager.cc:1428] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:45.118892 3531 tablet_bootstrap.cc:492] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d: No bootstrap required, opened a new log
I20260927 10:54:45.118969 3531 ts_tablet_manager.cc:1397] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:45.119308 3531 raft_consensus.cc:348] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.119406 3531 raft_consensus.cc:374] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:45.119443 3531 raft_consensus.cc:729] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaefa931d524ec1ac2156690015045d, State: Initialized, Role: FOLLOWER
I20260927 10:54:45.119549 3531 consensus_queue.cc:260] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.119961 3531 ts_tablet_manager.cc:1428] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:54:45.284446 3537 raft_consensus.cc:482] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:45.284600 3537 raft_consensus.cc:504] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.285111 3537 leader_election.cc:288] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263), b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515)
I20260927 10:54:45.285605 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9939721e696b43fd8d2b9cd809a1fab2" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aaefa931d524ec1ac2156690015045d" is_pre_election: true
I20260927 10:54:45.285619 3232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9939721e696b43fd8d2b9cd809a1fab2" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715" is_pre_election: true
I20260927 10:54:45.285776 3093 raft_consensus.cc:2454] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 0.
I20260927 10:54:45.285851 3232 raft_consensus.cc:2454] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 0.
I20260927 10:54:45.286180 3304 leader_election.cc:302] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaefa931d524ec1ac2156690015045d, 90a143649c3a479babf380f8e894640a; no voters:
I20260927 10:54:45.286399 3537 raft_consensus.cc:2781] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:45.286480 3537 raft_consensus.cc:482] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:45.286537 3537 raft_consensus.cc:3037] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:45.287544 3537 raft_consensus.cc:504] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.287855 3537 leader_election.cc:288] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 election: Requested vote from peers 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263), b25cd8be2c1c4ffdb11fba5047349715 (127.1.46.194:43515)
I20260927 10:54:45.288226 3232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9939721e696b43fd8d2b9cd809a1fab2" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25cd8be2c1c4ffdb11fba5047349715"
I20260927 10:54:45.288226 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9939721e696b43fd8d2b9cd809a1fab2" candidate_uuid: "90a143649c3a479babf380f8e894640a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aaefa931d524ec1ac2156690015045d"
I20260927 10:54:45.288358 3093 raft_consensus.cc:3037] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:45.288365 3232 raft_consensus.cc:3037] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:45.289245 3232 raft_consensus.cc:2454] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 1.
I20260927 10:54:45.289469 3093 raft_consensus.cc:2454] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a143649c3a479babf380f8e894640a in term 1.
I20260927 10:54:45.289558 3304 leader_election.cc:302] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a143649c3a479babf380f8e894640a, b25cd8be2c1c4ffdb11fba5047349715; no voters:
I20260927 10:54:45.289719 3537 raft_consensus.cc:2781] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:45.289891 3537 raft_consensus.cc:686] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 1 LEADER]: Becoming Leader. State: Replica: 90a143649c3a479babf380f8e894640a, State: Running, Role: LEADER
I20260927 10:54:45.290020 3537 consensus_queue.cc:237] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } }
I20260927 10:54:45.291750 2934 catalog_manager.cc:5495] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a143649c3a479babf380f8e894640a (127.1.46.195). New cstate: current_term: 1 leader_uuid: "90a143649c3a479babf380f8e894640a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a143649c3a479babf380f8e894640a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44197 } health_report { overall_health: HEALTHY } } }
I20260927 10:54:45.317984 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:45.314639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.1.46.195:44197)
0927 10:54:45.314953 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:45.314963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:45.314983 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:54:45.315021 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:45.315023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:45.315060 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:45.315153 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:45.315411 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:45.315955 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:45.316789 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:45.317061 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:45.317222 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:45.317502 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:45.317581 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:45.317666 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:45.317757 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 10:54:45.317805 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260927 10:54:45.324256 3232 raft_consensus.cc:1261] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Refusing update from remote peer 90a143649c3a479babf380f8e894640a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:45.324297 3093 raft_consensus.cc:1261] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d [term 1 FOLLOWER]: Refusing update from remote peer 90a143649c3a479babf380f8e894640a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:45.324700 3537 consensus_queue.cc:1035] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25cd8be2c1c4ffdb11fba5047349715" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:45.324888 3547 consensus_queue.cc:1035] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aaefa931d524ec1ac2156690015045d" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:54:45.338707 3502 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:54:46.890838 2935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:46.891011 2935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:46.893432 2935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06e4b63303ef47eb8099457a606cca1e: Sending DeleteTablet for 3 replicas of tablet 9939721e696b43fd8d2b9cd809a1fab2
I20260927 10:54:46.893922 3072 tablet_service.cc:1514] Processing DeleteTablet for tablet 9939721e696b43fd8d2b9cd809a1fab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:46 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:51420
I20260927 10:54:46.894052 3353 tablet_service.cc:1514] Processing DeleteTablet for tablet 9939721e696b43fd8d2b9cd809a1fab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:46 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:36544
I20260927 10:54:46.894085 3213 tablet_service.cc:1514] Processing DeleteTablet for tablet 9939721e696b43fd8d2b9cd809a1fab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:46 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44452
I20260927 10:54:46.894295 3576 tablet_replica.cc:330] T 9939721e696b43fd8d2b9cd809a1fab2 P 6aaefa931d524ec1ac2156690015045d: stopping tablet replica
I20260927 10:54:46.894348 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 3008
I20260927 10:54:46.898801 3578 tablet_replica.cc:330] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715: stopping tablet replica
I20260927 10:54:46.898948 3578 raft_consensus.cc:2229] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:46.899088 3578 raft_consensus.cc:2258] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:46.899329 3577 tablet_replica.cc:330] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: stopping tablet replica
I20260927 10:54:46.899478 3577 raft_consensus.cc:2229] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:46.899669 3577 raft_consensus.cc:2258] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:46.899722 3578 ts_tablet_manager.cc:1905] T 9939721e696b43fd8d2b9cd809a1fab2 P b25cd8be2c1c4ffdb11fba5047349715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:46.900261 3577 ts_tablet_manager.cc:1905] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:54:46.901350 2920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:36263 (error 108)
I20260927 10:54:46.901602 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 3148
I20260927 10:54:46.902792 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:46.902353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:36263 (local address 127.0.0.1:51946)
0927 10:54:46.902632 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:46.902711 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:36263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
W20260927 10:54:46.902928 2920 catalog_manager.cc:4584] TS 6aaefa931d524ec1ac2156690015045d (127.1.46.193:36263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9939721e696b43fd8d2b9cd809a1fab2: Network error: Client connection negotiation failed: client connection to 127.1.46.193:36263: connect: Connection refused (error 111)
I20260927 10:54:46.905391 3577 ts_tablet_manager.cc:1918] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:54:46.905476 3577 log.cc:1196] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/9939721e696b43fd8d2b9cd809a1fab2
I20260927 10:54:46.905771 3577 ts_tablet_manager.cc:1939] T 9939721e696b43fd8d2b9cd809a1fab2 P 90a143649c3a479babf380f8e894640a: Deleting consensus metadata
I20260927 10:54:46.906507 2923 catalog_manager.cc:4841] TS 90a143649c3a479babf380f8e894640a (127.1.46.195:44197): tablet 9939721e696b43fd8d2b9cd809a1fab2 (table test-table [id=a9d085555f714d30a7dc4ef669d4cf17]) successfully deleted
I20260927 10:54:46.909123 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 3288
I20260927 10:54:46.911098 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:46.909065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:43515 (local address 127.0.0.1:45202)
0927 10:54:46.909209 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:46.910990 (+ 1781us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.194:43515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:54:46.915846 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 2904
2026-09-27T10:54:46Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6643 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:46 dist-test-slave-cffh krb5kdc[3583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:46 dist-test-slave-cffh krb5kdc[3583](info): set up 2 sockets
Sep 27 10:54:46 dist-test-slave-cffh krb5kdc[3583](info): commencing operation
krb5kdc: starting...
W20260927 10:54:47.954407 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:48.073835 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:35319
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:48.180375 3596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:48.180670 3596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:48.180768 3596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:48.184823 3596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:48.184899 3596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:48.184935 3596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:48.184990 3596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:48.185050 3596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:48.190161 3596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:35319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:48.191177 3596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:48.192168 3596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:54:48.200680 3596 server_base.cc:1034] running on GCE node
W20260927 10:54:48.200629 3601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:48.200700 3602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:48.200641 3604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:48.201403 3596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:48.202409 3596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:48.203562 3596 hybrid_clock.cc:648] HybridClock initialized: now 1790506488203545 us; error 35 us; skew 500 ppm
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:48.206228 3596 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:48.207109 3596 webserver.cc:460] Webserver started at http://127.1.46.254:34245/ using document root <none> and password file <none>
I20260927 10:54:48.207645 3596 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:48.207732 3596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:48.207950 3596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:48.209892 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "ae186737281244f1bf7a1e02ae99c14d"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "9887a7bc51ba619d0614efe085ba9d6e"
server_key_iv: "ae032f03683ed3be4213c318f9ddcd65"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.210443 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "ae186737281244f1bf7a1e02ae99c14d"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "9887a7bc51ba619d0614efe085ba9d6e"
server_key_iv: "ae032f03683ed3be4213c318f9ddcd65"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.213862 3596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:48.216224 3611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.217327 3596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:48.217458 3596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "ae186737281244f1bf7a1e02ae99c14d"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "9887a7bc51ba619d0614efe085ba9d6e"
server_key_iv: "ae032f03683ed3be4213c318f9ddcd65"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.217577 3596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:48.233171 3596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:48.235756 3596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:48.235940 3596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:48.242328 3596 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:35319
I20260927 10:54:48.242391 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:35319 every 8 connection(s)
I20260927 10:54:48.243278 3596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:48.245831 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:48.249336 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 3596
I20260927 10:54:48.249446 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:48.249719 1211 external_mini_cluster.cc:1428] Setting key b2ad8d967b904bb72c3ec5caaf90b744
I20260927 10:54:48.251667 3664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d: Bootstrap starting.
I20260927 10:54:48.253736 3664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:48.254433 3664 log.cc:824] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:48.256439 3664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d: No bootstrap required, opened a new log
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:48.260468 3664 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae186737281244f1bf7a1e02ae99c14d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35319 } }
I20260927 10:54:48.260702 3664 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:48.260779 3664 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae186737281244f1bf7a1e02ae99c14d, State: Initialized, Role: FOLLOWER
I20260927 10:54:48.261232 3664 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae186737281244f1bf7a1e02ae99c14d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35319 } }
I20260927 10:54:48.261379 3664 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:48.261448 3664 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:48.261545 3664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:48.262385 3664 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae186737281244f1bf7a1e02ae99c14d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35319 } }
I20260927 10:54:48.262737 3664 leader_election.cc:302] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae186737281244f1bf7a1e02ae99c14d; no voters:
I20260927 10:54:48.263090 3664 leader_election.cc:288] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:48.263156 3669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:48.263360 3669 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [term 1 LEADER]: Becoming Leader. State: Replica: ae186737281244f1bf7a1e02ae99c14d, State: Running, Role: LEADER
I20260927 10:54:48.263713 3669 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae186737281244f1bf7a1e02ae99c14d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35319 } }
I20260927 10:54:48.264160 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:48.251119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.1.46.254:35319)
0927 10:54:48.251515 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:48.251530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:48.251564 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:54:48.252224 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:48.252240 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:48.252557 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:48.252881 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:48.253648 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.254791 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.255965 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.256418 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.256577 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:48.259266 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:48.259270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:48.259283 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:48.259308 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:48.261077 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:48.261547 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:48.261554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:48.261560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:48.261714 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:48.261992 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:48.261997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:48.261999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:48.262369 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:48.262612 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:48.262868 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 10:54:48.263200 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260927 10:54:48.264364 3664 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:48.265250 3670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae186737281244f1bf7a1e02ae99c14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae186737281244f1bf7a1e02ae99c14d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35319 } } }
I20260927 10:54:48.265372 3670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:48.265671 3671 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae186737281244f1bf7a1e02ae99c14d. Latest consensus state: current_term: 1 leader_uuid: "ae186737281244f1bf7a1e02ae99c14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae186737281244f1bf7a1e02ae99c14d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35319 } } }
I20260927 10:54:48.265756 3671 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:48.266625 3674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:48.269812 3674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:48.275435 3674 catalog_manager.cc:1347] Generated new cluster ID: 8f9b343ca97343c39b309e92685c6aa7
I20260927 10:54:48.275492 3674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:48.292738 3674 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:48.293982 3674 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:48.300990 3674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae186737281244f1bf7a1e02ae99c14d: Generated new TSK 0
I20260927 10:54:48.301739 3674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:48.338106 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:35319
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:48.477484 3690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:48.477764 3690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:48.477841 3690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:48.481806 3690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:48.481886 3690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:48.482008 3690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:48.487192 3690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:35319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:48.488157 3690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:48.489131 3690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:48.498080 3696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:48.498131 3698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:48.498080 3695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:48.498324 3690 server_base.cc:1034] running on GCE node
I20260927 10:54:48.498816 3690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:48.499405 3690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:48.500598 3690 hybrid_clock.cc:648] HybridClock initialized: now 1790506488500587 us; error 28 us; skew 500 ppm
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:48.503314 3690 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:48.504083 3690 webserver.cc:460] Webserver started at http://127.1.46.193:41091/ using document root <none> and password file <none>
I20260927 10:54:48.504573 3690 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:48.504629 3690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:48.504779 3690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:48.506673 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "a565f239034d405a8b524a9194db0f13"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "d309252d270b8e2a03373d1f132ac9b5"
server_key_iv: "a2c6591978a3b09b3f451102ad54f10f"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.507176 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "a565f239034d405a8b524a9194db0f13"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "d309252d270b8e2a03373d1f132ac9b5"
server_key_iv: "a2c6591978a3b09b3f451102ad54f10f"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.510504 3690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:54:48.512784 3705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.513900 3690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:48.514021 3690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "a565f239034d405a8b524a9194db0f13"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "d309252d270b8e2a03373d1f132ac9b5"
server_key_iv: "a2c6591978a3b09b3f451102ad54f10f"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.514122 3690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:48.530110 3690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:48.533385 3690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:48.533636 3690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:48.534233 3690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:48.535151 3690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:48.535202 3690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.535275 3690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:48.535311 3690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.544515 3690 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:43951
I20260927 10:54:48.544556 3818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:43951 every 8 connection(s)
I20260927 10:54:48.545352 3690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:48.555071 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 3690
I20260927 10:54:48.555187 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:48.555523 1211 external_mini_cluster.cc:1428] Setting key f9230f070d21a400291d17353900e39f
I20260927 10:54:48.559137 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:48.547010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:37233 (local address 127.1.46.254:35319)
0927 10:54:48.547168 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:48.547173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:48.548015 (+ 842us) server_negotiation.cc:407] Connection header received
0927 10:54:48.549397 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:48.549400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:48.549449 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:48.549532 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:48.550676 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.551217 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.552329 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.552587 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.552683 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:48.555020 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:48.555023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:48.555025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:48.555047 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:48.556712 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:48.557264 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:48.557267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:48.557268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:48.557339 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:48.557782 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:48.557786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:48.557788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:48.558105 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:48.558220 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:48.558858 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 10:54:48.558977 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:54:48.559855 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:48.547444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:35319 (local address 127.1.46.193:37233)
0927 10:54:48.547861 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:48.547906 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:48.549183 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:48.549668 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:48.549674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:48.550026 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:48.550469 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:48.550482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.551336 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.551339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:48.552200 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:48.552209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.552723 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.552726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:48.552854 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:48.553370 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:48.553388 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:48.554840 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:48.556825 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:48.556832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:48.556846 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:48.557163 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:48.557443 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:48.557446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:48.557448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:48.557626 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:48.558280 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:48.558287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:48.558612 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 10:54:48.558818 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 10:54:48.559113 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260927 10:54:48.560972 3819 heartbeater.cc:347] Connected to a master server at 127.1.46.254:35319
I20260927 10:54:48.561201 3819 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:48.561653 3819 heartbeater.cc:511] Master 127.1.46.254:35319 requested a full tablet report, sending...
I20260927 10:54:48.563277 3628 ts_manager.cc:194] Registered new tserver with Master: a565f239034d405a8b524a9194db0f13 (127.1.46.193:43951)
I20260927 10:54:48.564895 3628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:37233
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:48.585475 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:35319
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:48.705754 3824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:48.706002 3824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:48.706045 3824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:48.713500 3824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:48.713560 3824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:48.713652 3824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:48.721045 3824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:35319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:48.722353 3824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:48.723453 3824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:48.732705 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:48.732709 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:48.732883 3824 server_base.cc:1034] running on GCE node
W20260927 10:54:48.733150 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:48.733548 3824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:48.734172 3824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:48.735355 3824 hybrid_clock.cc:648] HybridClock initialized: now 1790506488735312 us; error 55 us; skew 500 ppm
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:48.738129 3824 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:48.739089 3824 webserver.cc:460] Webserver started at http://127.1.46.194:37631/ using document root <none> and password file <none>
I20260927 10:54:48.739660 3824 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:48.739746 3824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:48.739974 3824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:48.741945 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "005fbbc7e45243cb9054c6e4ae7907dd"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "74633c8a7eccfbde25ef86ee047b9da9"
server_key_iv: "5ffb40bf5caaba6a5ff39dc528a95ce9"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.742486 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "005fbbc7e45243cb9054c6e4ae7907dd"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "74633c8a7eccfbde25ef86ee047b9da9"
server_key_iv: "5ffb40bf5caaba6a5ff39dc528a95ce9"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.746001 3824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:48.748436 3839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.749655 3824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:48.749779 3824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "005fbbc7e45243cb9054c6e4ae7907dd"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "74633c8a7eccfbde25ef86ee047b9da9"
server_key_iv: "5ffb40bf5caaba6a5ff39dc528a95ce9"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.749879 3824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:48.764531 3824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:48.767755 3824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:48.767900 3824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:48.768390 3824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:48.769150 3824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:48.769196 3824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.769232 3824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:48.769246 3824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.777711 3824 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:34651
I20260927 10:54:48.777778 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:34651 every 8 connection(s)
I20260927 10:54:48.778654 3824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:48.781281 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 3824
I20260927 10:54:48.781433 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:48.781715 1211 external_mini_cluster.cc:1428] Setting key 5e4916a054e6d1f40fc5acc42e51b783
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:54:48.793285 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:48.780269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:59753 (local address 127.1.46.254:35319)
0927 10:54:48.780390 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:48.780393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:48.781146 (+ 753us) server_negotiation.cc:407] Connection header received
0927 10:54:48.782794 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:48.782797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:48.782837 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:48.782909 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:48.784104 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.784907 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.786479 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.786747 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.786857 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:48.789448 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:48.789452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:48.789455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:48.789483 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:48.791215 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:48.791730 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:48.791735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:48.791738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:48.791819 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:48.792155 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:48.792160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:48.792162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:48.792339 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:48.792443 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:48.793007 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 10:54:48.793122 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:54:48.794101 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:48.780596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:35319 (local address 127.1.46.194:59753)
0927 10:54:48.780996 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:48.781052 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:48.782571 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:48.783106 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:48.783115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:48.783498 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:48.783966 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:48.783978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.785050 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.785055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:48.786319 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:48.786327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.786885 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.786889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:48.787068 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:48.787651 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:48.787678 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:48.789293 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:48.791331 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:48.791336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:48.791346 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:48.791593 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:48.791919 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:48.791923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:48.791926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:48.792052 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:48.792444 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:48.792450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:48.792753 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 10:54:48.792939 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 10:54:48.793229 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
I20260927 10:54:48.795368 3953 heartbeater.cc:347] Connected to a master server at 127.1.46.254:35319
I20260927 10:54:48.795652 3953 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:48.796275 3953 heartbeater.cc:511] Master 127.1.46.254:35319 requested a full tablet report, sending...
I20260927 10:54:48.797494 3628 ts_manager.cc:194] Registered new tserver with Master: 005fbbc7e45243cb9054c6e4ae7907dd (127.1.46.194:34651)
I20260927 10:54:48.798154 3628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:59753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:48.810381 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:35319
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:48.918524 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:48.918835 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:48.918921 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:48.922982 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:48.923053 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:48.923139 3958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:48.928236 3958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:35319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:48.929260 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:48.930222 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:48.939091 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:48.939127 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:48.939132 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:48.939494 3958 server_base.cc:1034] running on GCE node
I20260927 10:54:48.939855 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:48.940457 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:48.941612 3958 hybrid_clock.cc:648] HybridClock initialized: now 1790506488941598 us; error 26 us; skew 500 ppm
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:48.944360 3958 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:48.945106 3958 webserver.cc:460] Webserver started at http://127.1.46.195:44621/ using document root <none> and password file <none>
I20260927 10:54:48.945668 3958 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:48.945726 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:48.945873 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:48.947708 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "3a7f9e19e48d45a9aa3534e1feec8a77"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "12da554aa39b6da4c7937e9127e9b193"
server_key_iv: "ba38d269dc86e8dd695c4c69948f7f7d"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.948185 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "3a7f9e19e48d45a9aa3534e1feec8a77"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "12da554aa39b6da4c7937e9127e9b193"
server_key_iv: "ba38d269dc86e8dd695c4c69948f7f7d"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.951704 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:48.953949 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.955194 3958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:48.955322 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "3a7f9e19e48d45a9aa3534e1feec8a77"
format_stamp: "Formatted at 2026-09-27 10:54:48 on dist-test-slave-cffh"
server_key: "12da554aa39b6da4c7937e9127e9b193"
server_key_iv: "ba38d269dc86e8dd695c4c69948f7f7d"
server_key_version: "encryptionkey@0"
I20260927 10:54:48.955422 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:48.976378 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:48.979655 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:48.979861 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:48.980450 3958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:48.981267 3958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:48.981340 3958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.981412 3958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:48.981455 3958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:48.990032 3958 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39373
I20260927 10:54:48.990068 4086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39373 every 8 connection(s)
I20260927 10:54:48.990896 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:48.996318 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 3958
I20260927 10:54:48.996438 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:48.996681 1211 external_mini_cluster.cc:1428] Setting key 38f07f6089b1478eedb954bb0dc39bb9
Sep 27 10:54:48 dist-test-slave-cffh krb5kdc[3583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:49.004240 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:48.992649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:45445 (local address 127.1.46.254:35319)
0927 10:54:48.992788 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:48.992792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:48.994052 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 10:54:48.994938 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:48.994941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:48.994988 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:48.995068 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:48.996188 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.996815 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.998243 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.998470 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.998618 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:49.000889 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:49.000893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:49.000897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:49.000929 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:49.002229 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:49.002780 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:49.002786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:49.002789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:49.002871 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:49.003207 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:49.003212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:49.003214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:49.003384 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:49.003502 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:49.003972 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 10:54:49.004080 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:54:49.004963 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:48.993041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:35319 (local address 127.1.46.195:45445)
0927 10:54:48.993895 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:48.993938 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:48.994746 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:48.995219 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:48.995226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:48.995572 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:48.996022 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:48.996035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.996933 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.996937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:48.998109 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:48.998120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:48.998641 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:48.998645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:48.998843 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:48.999348 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:48.999363 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:49.000736 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:49.002342 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:49.002348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:49.002363 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:49.002684 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:49.002976 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:49.002980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:49.002983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:49.003110 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:49.003503 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:49.003510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:49.003747 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 10:54:49.003918 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 10:54:49.004214 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":527,"threads_started":1}
I20260927 10:54:49.005944 4087 heartbeater.cc:347] Connected to a master server at 127.1.46.254:35319
I20260927 10:54:49.006172 4087 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:49.006675 4087 heartbeater.cc:511] Master 127.1.46.254:35319 requested a full tablet report, sending...
I20260927 10:54:49.007601 3628 ts_manager.cc:194] Registered new tserver with Master: 3a7f9e19e48d45a9aa3534e1feec8a77 (127.1.46.195:39373)
I20260927 10:54:49.008145 3628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:45445
I20260927 10:54:49.010484 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:54:49.023521 3667 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:49.021954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.1.46.254:35319)
0927 10:54:49.022102 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:49.022106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:49.022280 (+ 174us) server_negotiation.cc:407] Connection header received
0927 10:54:49.022451 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:49.022471 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:49.022518 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:49.022669 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:49.023353 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39858: BlockingRecv error: recv got EOF from 127.0.0.1:39858 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:54:49.024752 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 3690
I20260927 10:54:49.030524 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 3824
I20260927 10:54:49.035612 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 3958
I20260927 10:54:49.040416 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 3596
2026-09-27T10:54:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2122 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:49 dist-test-slave-cffh krb5kdc[4099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:49 dist-test-slave-cffh krb5kdc[4099](info): set up 2 sockets
Sep 27 10:54:49 dist-test-slave-cffh krb5kdc[4099](info): commencing operation
krb5kdc: starting...
W20260927 10:54:50.078536 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:50 dist-test-slave-cffh krb5kdc[4099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:50.197311 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43897
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:43897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:50.305459 4112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:50.305754 4112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:50.305835 4112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:50.309770 4112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:50.309855 4112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:50.309893 4112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:50.309932 4112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:50.309963 4112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:50.315100 4112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:43897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:50.316145 4112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:50.317165 4112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:50.325714 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:50.325716 4120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:50.325748 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:50.325960 4112 server_base.cc:1034] running on GCE node
I20260927 10:54:50.326529 4112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:50.327682 4112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:50.328914 4112 hybrid_clock.cc:648] HybridClock initialized: now 1790506490328878 us; error 48 us; skew 500 ppm
Sep 27 10:54:50 dist-test-slave-cffh krb5kdc[4099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506490, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:50.338302 4112 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:50.339216 4112 webserver.cc:460] Webserver started at http://127.1.46.254:33651/ using document root <none> and password file <none>
I20260927 10:54:50.339758 4112 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:50.339847 4112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:50.340080 4112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:50.342029 4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38"
format_stamp: "Formatted at 2026-09-27 10:54:50 on dist-test-slave-cffh"
server_key: "4783de2be7042aa0bb6512262861153e"
server_key_iv: "868b840ac5fef5aeac524eeee967640e"
server_key_version: "encryptionkey@0"
I20260927 10:54:50.342650 4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38"
format_stamp: "Formatted at 2026-09-27 10:54:50 on dist-test-slave-cffh"
server_key: "4783de2be7042aa0bb6512262861153e"
server_key_iv: "868b840ac5fef5aeac524eeee967640e"
server_key_version: "encryptionkey@0"
I20260927 10:54:50.346127 4112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:50.348423 4127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:50.349560 4112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:54:50.349696 4112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38"
format_stamp: "Formatted at 2026-09-27 10:54:50 on dist-test-slave-cffh"
server_key: "4783de2be7042aa0bb6512262861153e"
server_key_iv: "868b840ac5fef5aeac524eeee967640e"
server_key_version: "encryptionkey@0"
I20260927 10:54:50.349802 4112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:50.369688 4112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:50.372956 4112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:50.373157 4112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:50.379796 4112 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:43897
I20260927 10:54:50.380119 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:43897 every 8 connection(s)
I20260927 10:54:50.380716 4112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:50.383038 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4112
I20260927 10:54:50.383165 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:50.383433 1211 external_mini_cluster.cc:1428] Setting key 6da9f401cd2e008a914f380c024b3f14
I20260927 10:54:50.383517 4180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:50.390270 4180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38: Bootstrap starting.
Sep 27 10:54:50 dist-test-slave-cffh krb5kdc[4099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:50.393079 4180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:50.393978 4180 log.cc:824] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:50.396261 4180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38: No bootstrap required, opened a new log
I20260927 10:54:50.398913 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:50.384699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.1.46.254:43897)
0927 10:54:50.385097 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:50.385111 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:50.385145 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:54:50.386116 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:50.386134 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:50.386450 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:50.386788 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:50.387679 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:50.388636 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:50.389817 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:50.390289 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:50.390440 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:50.393401 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:50.393407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:50.393422 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:50.393452 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:50.395657 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:50.396101 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:50.396107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:50.396113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:50.396253 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:50.396586 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:50.396590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:50.396592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:50.397085 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:50.397296 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:50.397551 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 10:54:50.397835 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
I20260927 10:54:50.400781 4180 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43897 } }
I20260927 10:54:50.401065 4180 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:50.401127 4180 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c7a8bc8fa34c60a0dd384a92da9e38, State: Initialized, Role: FOLLOWER
I20260927 10:54:50.401593 4180 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43897 } }
I20260927 10:54:50.401757 4180 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:50.401834 4180 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:50.401937 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:50.401939 4180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:50.403152 4180 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43897 } }
I20260927 10:54:50.403523 4180 leader_election.cc:302] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c7a8bc8fa34c60a0dd384a92da9e38; no voters:
I20260927 10:54:50.403899 4180 leader_election.cc:288] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:50.404101 4185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:50.404363 4185 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [term 1 LEADER]: Becoming Leader. State: Replica: e8c7a8bc8fa34c60a0dd384a92da9e38, State: Running, Role: LEADER
I20260927 10:54:50.404744 4185 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43897 } }
I20260927 10:54:50.405193 4180 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:50.405947 4185 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c7a8bc8fa34c60a0dd384a92da9e38. Latest consensus state: current_term: 1 leader_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43897 } } }
I20260927 10:54:50.406070 4185 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:50.406734 4188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7a8bc8fa34c60a0dd384a92da9e38" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43897 } } }
I20260927 10:54:50.406847 4188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:50.408146 4194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:50.411085 4194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:50.416306 4194 catalog_manager.cc:1347] Generated new cluster ID: 98fcef75aec04aab95d4d35a2a327bcd
I20260927 10:54:50.416371 4194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:50.421725 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:50.416536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.1.46.254:43897)
0927 10:54:50.416667 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:50.416672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:50.417033 (+ 361us) server_negotiation.cc:407] Connection header received
0927 10:54:50.417213 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:50.417217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:50.417274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:50.417363 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:50.417935 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:50.418472 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:50.419394 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:50.419645 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:50.419775 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:50.420783 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:50.420788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:50.420794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:50.421211 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:50.421240 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:50.421257 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:50.421363 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:50.421510 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:54:50.421573 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:54:50.424904 4194 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:50.426097 4194 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:50.429675 4194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c7a8bc8fa34c60a0dd384a92da9e38: Generated new TSK 0
I20260927 10:54:50.430428 4194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:50.442273 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4112
2026-09-27T10:54:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1399 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:50 dist-test-slave-cffh krb5kdc[4213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:50 dist-test-slave-cffh krb5kdc[4213](info): set up 2 sockets
Sep 27 10:54:50 dist-test-slave-cffh krb5kdc[4213](info): commencing operation
krb5kdc: starting...
W20260927 10:54:51.486665 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:51 dist-test-slave-cffh krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:51.607581 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:42247
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:51.718777 4226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:51.719065 4226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:51.719147 4226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:51.723162 4226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:51.723237 4226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:51.723272 4226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:51.723327 4226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:51.723366 4226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:51.728437 4226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:51.729393 4226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:51.730326 4226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:51.738294 4234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:51.738319 4232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:51.738616 4226 server_base.cc:1034] running on GCE node
W20260927 10:54:51.738683 4231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:51.739219 4226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:51.740190 4226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:51.741389 4226 hybrid_clock.cc:648] HybridClock initialized: now 1790506491741348 us; error 50 us; skew 500 ppm
Sep 27 10:54:51 dist-test-slave-cffh krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:51.744169 4226 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:51.744952 4226 webserver.cc:460] Webserver started at http://127.1.46.254:46467/ using document root <none> and password file <none>
I20260927 10:54:51.745455 4226 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:51.745540 4226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:51.745775 4226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:51.747735 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "614d397d59654dc19b03a7269044b3ba"
format_stamp: "Formatted at 2026-09-27 10:54:51 on dist-test-slave-cffh"
server_key: "8e08e6b5d568da2ee6c919a27faa9e2e"
server_key_iv: "afea7ac4b54b1bd683b02677fab52881"
server_key_version: "encryptionkey@0"
I20260927 10:54:51.748271 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "614d397d59654dc19b03a7269044b3ba"
format_stamp: "Formatted at 2026-09-27 10:54:51 on dist-test-slave-cffh"
server_key: "8e08e6b5d568da2ee6c919a27faa9e2e"
server_key_iv: "afea7ac4b54b1bd683b02677fab52881"
server_key_version: "encryptionkey@0"
I20260927 10:54:51.751683 4226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:54:51.753876 4241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:51.755024 4226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:51.755174 4226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "614d397d59654dc19b03a7269044b3ba"
format_stamp: "Formatted at 2026-09-27 10:54:51 on dist-test-slave-cffh"
server_key: "8e08e6b5d568da2ee6c919a27faa9e2e"
server_key_iv: "afea7ac4b54b1bd683b02677fab52881"
server_key_version: "encryptionkey@0"
I20260927 10:54:51.755278 4226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:51.777634 4226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:51.780828 4226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:51.781059 4226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:51.787845 4226 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:42247
I20260927 10:54:51.788160 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:42247 every 8 connection(s)
I20260927 10:54:51.788797 4226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:51.791599 4294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:51.793300 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4226
I20260927 10:54:51.793421 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:51.793713 1211 external_mini_cluster.cc:1428] Setting key a422cc9fff42f004cce333885580b404
I20260927 10:54:51.798301 4294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba: Bootstrap starting.
I20260927 10:54:51.801014 4294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:51.802062 4294 log.cc:824] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:54:51 dist-test-slave-cffh krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:51.804330 4294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba: No bootstrap required, opened a new log
I20260927 10:54:51.808833 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:51.795050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.1.46.254:42247)
0927 10:54:51.795419 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:51.795434 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:51.795468 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:54:51.796082 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:51.796095 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:51.796391 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:51.796658 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:51.797647 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:51.798688 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:51.799927 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:51.800372 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:51.800483 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:51.803691 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:51.803696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:51.803728 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:51.803758 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:51.805640 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:51.806124 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:51.806130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:51.806136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:51.806294 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:51.806619 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:51.806623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:51.806625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:51.807191 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:51.807377 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:51.807601 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 10:54:51.807888 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":118,"threads_started":1}
I20260927 10:54:51.808993 4294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614d397d59654dc19b03a7269044b3ba" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 42247 } }
I20260927 10:54:51.809276 4294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:51.809336 4294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614d397d59654dc19b03a7269044b3ba, State: Initialized, Role: FOLLOWER
I20260927 10:54:51.809753 4294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614d397d59654dc19b03a7269044b3ba" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 42247 } }
I20260927 10:54:51.809930 4294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:51.810002 4294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:51.810101 4294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:51.811069 4294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614d397d59654dc19b03a7269044b3ba" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 42247 } }
I20260927 10:54:51.811414 4294 leader_election.cc:302] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 614d397d59654dc19b03a7269044b3ba; no voters:
I20260927 10:54:51.811717 4294 leader_election.cc:288] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:51.811823 4299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:51.812057 4299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [term 1 LEADER]: Becoming Leader. State: Replica: 614d397d59654dc19b03a7269044b3ba, State: Running, Role: LEADER
I20260927 10:54:51.812382 4299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614d397d59654dc19b03a7269044b3ba" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 42247 } }
I20260927 10:54:51.812865 4294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:51.813947 4301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 614d397d59654dc19b03a7269044b3ba. Latest consensus state: current_term: 1 leader_uuid: "614d397d59654dc19b03a7269044b3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614d397d59654dc19b03a7269044b3ba" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 42247 } } }
I20260927 10:54:51.814083 4301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:51.814347 4300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "614d397d59654dc19b03a7269044b3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614d397d59654dc19b03a7269044b3ba" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 42247 } } }
I20260927 10:54:51.814450 4300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:51.814536 4308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:51.817433 4308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:51.822636 4308 catalog_manager.cc:1347] Generated new cluster ID: 005b85585ffc4c57b22f8853fb2316f0
I20260927 10:54:51.822695 4308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:51.850234 4308 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:51.851193 4308 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:51.858965 4308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 614d397d59654dc19b03a7269044b3ba: Generated new TSK 0
I20260927 10:54:51.859769 4308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:51.893466 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:42247
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:52.005419 4320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:52.005642 4320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:52.005681 4320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:52.009658 4320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:52.009704 4320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:52.009795 4320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:52.015158 4320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:52.016094 4320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:52.017046 4320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:52.026082 4328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:52.026086 4326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:52.026074 4325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:52.026391 4320 server_base.cc:1034] running on GCE node
I20260927 10:54:52.026818 4320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:52.027391 4320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:52.028554 4320 hybrid_clock.cc:648] HybridClock initialized: now 1790506492028538 us; error 28 us; skew 500 ppm
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:52.031178 4320 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:52.031949 4320 webserver.cc:460] Webserver started at http://127.1.46.193:40007/ using document root <none> and password file <none>
I20260927 10:54:52.032573 4320 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:52.032657 4320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:52.032860 4320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:52.034839 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "5736ec3671b6479f930cd77a24f22fac"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "4c670e1590be5579192e5f253ec3de00"
server_key_iv: "30b3acec9b0546e8b2190bfa2ad2e433"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.035318 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "5736ec3671b6479f930cd77a24f22fac"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "4c670e1590be5579192e5f253ec3de00"
server_key_iv: "30b3acec9b0546e8b2190bfa2ad2e433"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.038733 4320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:52.040877 4335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.042011 4320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:52.042096 4320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "5736ec3671b6479f930cd77a24f22fac"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "4c670e1590be5579192e5f253ec3de00"
server_key_iv: "30b3acec9b0546e8b2190bfa2ad2e433"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.042169 4320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:52.056788 4320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:52.059495 4320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:52.059629 4320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:52.060086 4320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:52.060822 4320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:52.060868 4320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.060904 4320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:52.060956 4320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.069332 4320 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:43539
I20260927 10:54:52.069418 4448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:43539 every 8 connection(s)
I20260927 10:54:52.070187 4320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:52.079540 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4320
I20260927 10:54:52.079691 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:52.079990 1211 external_mini_cluster.cc:1428] Setting key 664d243fba947f533304750f14e9f42a
I20260927 10:54:52.083951 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:52.071879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:42023 (local address 127.1.46.254:42247)
0927 10:54:52.072027 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:52.072032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:52.072910 (+ 878us) server_negotiation.cc:407] Connection header received
0927 10:54:52.074220 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:52.074223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:52.074269 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:52.074342 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:52.075517 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.076051 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.077110 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.077336 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.077418 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:52.079795 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:52.079799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:52.079801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:52.079824 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:52.081539 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.082139 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.082142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.082143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.082227 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.082654 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.082656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.082658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.082838 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:52.082922 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:52.083535 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 10:54:52.083699 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:54:52.084506 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:52.072321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:42247 (local address 127.1.46.193:42023)
0927 10:54:52.072748 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:52.072808 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:52.074005 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:52.074513 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:52.074519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:52.074921 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:52.075373 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:52.075385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.076154 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.076156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:52.076999 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:52.077006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.077460 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.077462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:52.077605 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:52.078114 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:52.078129 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:52.079609 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:52.081658 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:52.081664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:52.081678 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:52.082033 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:52.082324 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:52.082330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:52.082335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:52.082516 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:52.082945 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:52.082953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:52.083265 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 10:54:52.083425 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 10:54:52.083667 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260927 10:54:52.085984 4449 heartbeater.cc:347] Connected to a master server at 127.1.46.254:42247
I20260927 10:54:52.086211 4449 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:52.086714 4449 heartbeater.cc:511] Master 127.1.46.254:42247 requested a full tablet report, sending...
I20260927 10:54:52.088370 4258 ts_manager.cc:194] Registered new tserver with Master: 5736ec3671b6479f930cd77a24f22fac (127.1.46.193:43539)
I20260927 10:54:52.090054 4258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:42023
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:52.107301 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:42247
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:52.213513 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:52.213860 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:52.213946 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:52.217906 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:52.217983 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:52.218096 4454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:52.223078 4454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:52.223975 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:52.224888 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:52.233798 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:52.233819 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:52.233821 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:52.234144 4454 server_base.cc:1034] running on GCE node
I20260927 10:54:52.234535 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:52.235148 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:52.236333 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790506492236306 us; error 37 us; skew 500 ppm
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:52.238981 4454 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:52.239789 4454 webserver.cc:460] Webserver started at http://127.1.46.194:38833/ using document root <none> and password file <none>
I20260927 10:54:52.240319 4454 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:52.240401 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:52.240615 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:52.242658 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "ac0b4caa165d41bda83e412f04ca8139"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "5d9868bb945c054303cf7eab94227078"
server_key_iv: "8ea32bc7ad6cc113e8609999eb99a5bc"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.243199 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "ac0b4caa165d41bda83e412f04ca8139"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "5d9868bb945c054303cf7eab94227078"
server_key_iv: "8ea32bc7ad6cc113e8609999eb99a5bc"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.246786 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 10:54:52.249153 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.250286 4454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:52.250411 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "ac0b4caa165d41bda83e412f04ca8139"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "5d9868bb945c054303cf7eab94227078"
server_key_iv: "8ea32bc7ad6cc113e8609999eb99a5bc"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.250519 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:52.264334 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:52.267238 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:52.267431 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:52.267982 4454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:52.268756 4454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:52.268826 4454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.268894 4454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:52.268945 4454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.277184 4454 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:38745
I20260927 10:54:52.277206 4582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:38745 every 8 connection(s)
I20260927 10:54:52.277989 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:52.282678 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4454
I20260927 10:54:52.282811 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:52.283082 1211 external_mini_cluster.cc:1428] Setting key 77b24291be762f6929e55481be085a52
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:52.293181 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:52.279677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:43961 (local address 127.1.46.254:42247)
0927 10:54:52.279822 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:52.279826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:52.280557 (+ 731us) server_negotiation.cc:407] Connection header received
0927 10:54:52.281857 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:52.281860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:52.281907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:52.281988 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:52.283357 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.284001 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.285614 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.286005 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.286164 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:52.289095 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:52.289098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:52.289100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:52.289122 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:52.290775 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.291472 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.291475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.291476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.291561 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.291974 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.291976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.291978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.292188 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:52.292275 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:52.292932 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 10:54:52.293064 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:54:52.294397 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:52.280006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:42247 (local address 127.1.46.194:43961)
0927 10:54:52.280410 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:52.280448 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:52.281644 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:52.282177 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:52.282183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:52.282518 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:52.283193 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:52.283214 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.284137 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.284142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:52.285388 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:52.285399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.286218 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.286245 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:52.286464 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:52.287152 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:52.287170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:52.288904 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:52.290893 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:52.290902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:52.290919 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:52.291314 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:52.291656 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:52.291660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:52.291663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:52.291842 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:52.292316 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:52.292323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:52.292664 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 10:54:52.292881 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 10:54:52.293224 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
I20260927 10:54:52.295714 4583 heartbeater.cc:347] Connected to a master server at 127.1.46.254:42247
I20260927 10:54:52.295951 4583 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
I20260927 10:54:52.296447 4583 heartbeater.cc:511] Master 127.1.46.254:42247 requested a full tablet report, sending...
I20260927 10:54:52.297528 4258 ts_manager.cc:194] Registered new tserver with Master: ac0b4caa165d41bda83e412f04ca8139 (127.1.46.194:38745)
I20260927 10:54:52.298350 4258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:43961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:52.313108 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:42247
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:52.421124 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:52.421357 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:52.421396 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:52.425565 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:52.425613 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:52.425693 4588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:52.430954 4588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:52.431854 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:52.432806 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:52.442024 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:52.442039 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:52.442116 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:52.442529 4588 server_base.cc:1034] running on GCE node
I20260927 10:54:52.442975 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:52.443608 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:52.444797 4588 hybrid_clock.cc:648] HybridClock initialized: now 1790506492444781 us; error 34 us; skew 500 ppm
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:52.447759 4588 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:52.448649 4588 webserver.cc:460] Webserver started at http://127.1.46.195:39373/ using document root <none> and password file <none>
I20260927 10:54:52.449185 4588 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:52.449275 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:52.449517 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:52.451560 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "5839911956ea44e09a9b4300189f42b3"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "9e02ec1c70bb296813b71e454d7467aa"
server_key_iv: "1084fb0532ea30d22db0c0100c2db574"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.452112 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "5839911956ea44e09a9b4300189f42b3"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "9e02ec1c70bb296813b71e454d7467aa"
server_key_iv: "1084fb0532ea30d22db0c0100c2db574"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.455973 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:54:52.458523 4603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.459753 4588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:52.459878 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "5839911956ea44e09a9b4300189f42b3"
format_stamp: "Formatted at 2026-09-27 10:54:52 on dist-test-slave-cffh"
server_key: "9e02ec1c70bb296813b71e454d7467aa"
server_key_iv: "1084fb0532ea30d22db0c0100c2db574"
server_key_version: "encryptionkey@0"
I20260927 10:54:52.459980 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:52.504478 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:52.507829 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:52.508049 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:52.508755 4588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:52.509603 4588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:52.509671 4588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.509737 4588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:52.509768 4588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:52.518424 4588 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:41053
I20260927 10:54:52.518448 4716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:41053 every 8 connection(s)
I20260927 10:54:52.519392 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:52.529524 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4588
I20260927 10:54:52.529645 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:52.529937 1211 external_mini_cluster.cc:1428] Setting key b428c6365a910342399d346f675e4d80
I20260927 10:54:52.532927 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:52.521117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:49071 (local address 127.1.46.254:42247)
0927 10:54:52.521250 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:52.521253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:52.521998 (+ 745us) server_negotiation.cc:407] Connection header received
0927 10:54:52.523396 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:52.523400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:52.523457 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:52.523564 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:52.524667 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.525190 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.526291 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.526513 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.526652 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:52.529068 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:52.529071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:52.529073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:52.529097 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:52.530845 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.531445 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.531465 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.531467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.531571 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.531875 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.531879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.531881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.532117 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:52.532245 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:52.532726 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 10:54:52.532819 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:54:52.533676 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:52.521463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:42247 (local address 127.1.46.195:49071)
0927 10:54:52.521859 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:52.521897 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:52.523186 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:52.523752 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:52.523759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:52.524085 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:52.524510 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:52.524522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.525313 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.525317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:52.526166 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:52.526174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.526678 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.526682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:52.526851 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:52.527328 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:52.527343 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:52.528887 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:52.530936 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:52.530941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:52.530954 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:52.531337 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:52.531665 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:52.531667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:52.531668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:52.531783 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:52.532248 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:52.532256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:52.532489 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 10:54:52.532640 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:54:52.532856 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
I20260927 10:54:52.534798 4717 heartbeater.cc:347] Connected to a master server at 127.1.46.254:42247
I20260927 10:54:52.535002 4717 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:52.535418 4717 heartbeater.cc:511] Master 127.1.46.254:42247 requested a full tablet report, sending...
I20260927 10:54:52.536372 4258 ts_manager.cc:194] Registered new tserver with Master: 5839911956ea44e09a9b4300189f42b3 (127.1.46.195:41053)
I20260927 10:54:52.536906 4258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:49071
I20260927 10:54:52.543934 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506492, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:52.569766 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:52.560746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.1.46.254:42247)
0927 10:54:52.560862 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:52.560866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:52.561120 (+ 254us) server_negotiation.cc:407] Connection header received
0927 10:54:52.561226 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:52.561229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:52.561266 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:52.561335 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:52.562056 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.562544 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.563539 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:52.563784 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:52.563905 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:52.566634 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:52.566644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:52.566646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:52.566670 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:52.567893 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.568410 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.568416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.568419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.568511 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:52.568811 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:52.568815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:52.568818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:52.569049 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:52.569204 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:52.569449 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 10:54:52.569566 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 10:54:52.570128 4258 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39024
I20260927 10:54:52.570955 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4320
I20260927 10:54:52.576678 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4454
I20260927 10:54:52.581666 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4588
I20260927 10:54:52.586665 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4226
2026-09-27T10:54:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2146 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4729](info): set up 2 sockets
Sep 27 10:54:52 dist-test-slave-cffh krb5kdc[4729](info): commencing operation
krb5kdc: starting...
W20260927 10:54:53.618932 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:53 dist-test-slave-cffh krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:53.741329 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45067
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:53.849387 4742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:53.849614 4742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:53.849651 4742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:53.853572 4742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:53.853626 4742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:53.853644 4742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:53.853660 4742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:53.853703 4742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:53.858875 4742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:53.859798 4742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:53.860855 4742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:53.869050 4750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:53.869045 4748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:53.869042 4747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:53.869287 4742 server_base.cc:1034] running on GCE node
I20260927 10:54:53.869939 4742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:53.871066 4742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:53.872272 4742 hybrid_clock.cc:648] HybridClock initialized: now 1790506493872249 us; error 33 us; skew 500 ppm
Sep 27 10:54:53 dist-test-slave-cffh krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:53.875044 4742 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:53.875867 4742 webserver.cc:460] Webserver started at http://127.1.46.254:34375/ using document root <none> and password file <none>
I20260927 10:54:53.876403 4742 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:53.876493 4742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:53.876750 4742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:53.878767 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "06cbb2f666954b23a08f11117ee12f90"
format_stamp: "Formatted at 2026-09-27 10:54:53 on dist-test-slave-cffh"
server_key: "71e38159c9cb21c635358c4d5051abd9"
server_key_iv: "98ba167a614d4d537230c2e8a89624c7"
server_key_version: "encryptionkey@0"
I20260927 10:54:53.879333 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "06cbb2f666954b23a08f11117ee12f90"
format_stamp: "Formatted at 2026-09-27 10:54:53 on dist-test-slave-cffh"
server_key: "71e38159c9cb21c635358c4d5051abd9"
server_key_iv: "98ba167a614d4d537230c2e8a89624c7"
server_key_version: "encryptionkey@0"
I20260927 10:54:53.882818 4742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:54:53.885279 4757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:53.886479 4742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:53.886667 4742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "06cbb2f666954b23a08f11117ee12f90"
format_stamp: "Formatted at 2026-09-27 10:54:53 on dist-test-slave-cffh"
server_key: "71e38159c9cb21c635358c4d5051abd9"
server_key_iv: "98ba167a614d4d537230c2e8a89624c7"
server_key_version: "encryptionkey@0"
I20260927 10:54:53.886775 4742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:53.900038 4742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:53.902681 4742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:53.902874 4742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:53.909902 4742 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45067
I20260927 10:54:53.909917 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45067 every 8 connection(s)
I20260927 10:54:53.910846 4742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:53.913507 4810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:53.917531 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4742
I20260927 10:54:53.917647 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:53.917904 1211 external_mini_cluster.cc:1428] Setting key 5bc9ab73e3e10bec1f1fa6677a7b81f3
I20260927 10:54:53.919466 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90: Bootstrap starting.
I20260927 10:54:53.921777 4810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:53.922466 4810 log.cc:824] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:53.924249 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90: No bootstrap required, opened a new log
Sep 27 10:54:53 dist-test-slave-cffh krb5kdc[4729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:53.927596 4810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cbb2f666954b23a08f11117ee12f90" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45067 } }
I20260927 10:54:53.927832 4810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:53.927932 4810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cbb2f666954b23a08f11117ee12f90, State: Initialized, Role: FOLLOWER
I20260927 10:54:53.928400 4810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cbb2f666954b23a08f11117ee12f90" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45067 } }
I20260927 10:54:53.928555 4810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:53.928627 4810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:53.928722 4810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:53.929646 4810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cbb2f666954b23a08f11117ee12f90" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45067 } }
I20260927 10:54:53.929998 4810 leader_election.cc:302] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cbb2f666954b23a08f11117ee12f90; no voters:
I20260927 10:54:53.930299 4810 leader_election.cc:288] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:53.930392 4815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:53.930691 4815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [term 1 LEADER]: Becoming Leader. State: Replica: 06cbb2f666954b23a08f11117ee12f90, State: Running, Role: LEADER
I20260927 10:54:53.931154 4815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cbb2f666954b23a08f11117ee12f90" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45067 } }
I20260927 10:54:53.931645 4810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:53.931869 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:53.919213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.1.46.254:45067)
0927 10:54:53.919602 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:53.919624 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:53.919666 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:54:53.920275 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:53.920290 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:53.920590 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:53.920852 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:53.921742 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:53.922869 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:53.923759 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:53.924208 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:53.924370 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:53.926833 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:53.926837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:53.926849 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:53.926873 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:53.928867 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:53.929270 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:53.929275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:53.929281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:53.929420 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:53.929719 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:53.929722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:53.929724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:53.930193 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:53.930386 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:53.930732 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 10:54:53.930998 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":140,"threads_started":1}
I20260927 10:54:53.932200 4815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cbb2f666954b23a08f11117ee12f90. Latest consensus state: current_term: 1 leader_uuid: "06cbb2f666954b23a08f11117ee12f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cbb2f666954b23a08f11117ee12f90" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45067 } } }
I20260927 10:54:53.932236 4816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cbb2f666954b23a08f11117ee12f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cbb2f666954b23a08f11117ee12f90" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45067 } } }
I20260927 10:54:53.932403 4816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:53.932750 4815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:53.932888 4822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:53.936010 4822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:53.940698 4822 catalog_manager.cc:1347] Generated new cluster ID: 12b8a595ac9649c694ca7637fb8935a2
I20260927 10:54:53.940764 4822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:53.957538 4822 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:53.958766 4822 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:53.976434 4822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06cbb2f666954b23a08f11117ee12f90: Generated new TSK 0
I20260927 10:54:53.977063 4822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:54.013980 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45067
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:54.127417 4836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:54.127682 4836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:54.127774 4836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:54.131696 4836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:54.131773 4836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:54.131910 4836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:54.137075 4836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:54.138003 4836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:54.139061 4836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:54.148226 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:54.148286 4836 server_base.cc:1034] running on GCE node
W20260927 10:54:54.148238 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:54.148226 4844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:54.148849 4836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:54.149467 4836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:54.150665 4836 hybrid_clock.cc:648] HybridClock initialized: now 1790506494150653 us; error 23 us; skew 500 ppm
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:54.153381 4836 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:54.154168 4836 webserver.cc:460] Webserver started at http://127.1.46.193:42441/ using document root <none> and password file <none>
I20260927 10:54:54.154755 4836 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:54.154844 4836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:54.155061 4836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:54.156919 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "2a9a1dcb56774d1993a25205c0dfc7de"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "cfd3299bd8ffbdecf7ae4b440e07bf3c"
server_key_iv: "0e88f888a9783677e91797cf80cbc82e"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.157459 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "2a9a1dcb56774d1993a25205c0dfc7de"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "cfd3299bd8ffbdecf7ae4b440e07bf3c"
server_key_iv: "0e88f888a9783677e91797cf80cbc82e"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.161074 4836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:54:54.163431 4851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.164565 4836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:54.164700 4836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "2a9a1dcb56774d1993a25205c0dfc7de"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "cfd3299bd8ffbdecf7ae4b440e07bf3c"
server_key_iv: "0e88f888a9783677e91797cf80cbc82e"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.164801 4836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:54.194156 4836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:54.197278 4836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:54.197468 4836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:54.198020 4836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:54.198845 4836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:54.198910 4836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.198979 4836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:54.199030 4836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.207803 4836 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:46509
I20260927 10:54:54.207828 4964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:46509 every 8 connection(s)
I20260927 10:54:54.208702 4836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:54.210433 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4836
I20260927 10:54:54.210558 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:54.210888 1211 external_mini_cluster.cc:1428] Setting key e5f903b1f2d597c6dd84616e242d9516
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:54:54.223969 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.210430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:44399 (local address 127.1.46.254:45067)
0927 10:54:54.210618 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:54.210624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:54.211517 (+ 893us) server_negotiation.cc:407] Connection header received
0927 10:54:54.213502 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:54.213509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:54.213578 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:54.213741 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:54.215148 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.215717 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.216880 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.217123 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.217254 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.219804 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:54.219808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:54.219810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:54.219832 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:54.221459 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.222060 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.222066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.222068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.222192 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.222625 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.222629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.222632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.222814 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:54.222897 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:54.223586 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 10:54:54.223737 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:54:54.224702 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.210865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45067 (local address 127.1.46.193:44399)
0927 10:54:54.211368 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:54.211415 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:54.212733 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:54.214067 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:54.214074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:54.214426 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:54.214978 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:54.214992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.215862 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.215867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:54.216746 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:54.216755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.217289 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.217293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:54.217466 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.218081 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:54.218098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:54.219593 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:54.221608 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:54.221620 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:54.221634 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:54.221930 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:54.222296 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:54.222325 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:54.222328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:54.222475 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:54.222940 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:54.222949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:54.223261 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 10:54:54.223466 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 10:54:54.223846 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260927 10:54:54.225958 4965 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45067
I20260927 10:54:54.226217 4965 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:54.226727 4965 heartbeater.cc:511] Master 127.1.46.254:45067 requested a full tablet report, sending...
I20260927 10:54:54.228344 4774 ts_manager.cc:194] Registered new tserver with Master: 2a9a1dcb56774d1993a25205c0dfc7de (127.1.46.193:46509)
I20260927 10:54:54.230048 4774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:44399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:54.237519 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45067
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:54.344892 4970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:54.345101 4970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:54.345139 4970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:54.349136 4970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:54.349190 4970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:54.349272 4970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:54.354667 4970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:54.355842 4970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:54.357125 4970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:54.367797 4978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:54.367813 4976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:54.367813 4975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:54.368160 4970 server_base.cc:1034] running on GCE node
I20260927 10:54:54.368695 4970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:54.369236 4970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:54.370398 4970 hybrid_clock.cc:648] HybridClock initialized: now 1790506494370360 us; error 51 us; skew 500 ppm
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:54.373142 4970 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:54.373934 4970 webserver.cc:460] Webserver started at http://127.1.46.194:35745/ using document root <none> and password file <none>
I20260927 10:54:54.374456 4970 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:54.374514 4970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:54.374810 4970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:54.376755 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "9b9ed40b3ec945839b990b68ff0cad1e"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "805da5831bb2821f22ee871f08f03834"
server_key_iv: "5d0d2f1a6431bc6211c159a9ae0cc6ce"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.377317 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "9b9ed40b3ec945839b990b68ff0cad1e"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "805da5831bb2821f22ee871f08f03834"
server_key_iv: "5d0d2f1a6431bc6211c159a9ae0cc6ce"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.380977 4970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:54:54.383361 4985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.384543 4970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:54:54.384675 4970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "9b9ed40b3ec945839b990b68ff0cad1e"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "805da5831bb2821f22ee871f08f03834"
server_key_iv: "5d0d2f1a6431bc6211c159a9ae0cc6ce"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.384774 4970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:54.398773 4970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:54.401325 4970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:54.401528 4970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:54.402143 4970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:54.403024 4970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:54.403095 4970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.403154 4970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:54.403208 4970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.412750 4970 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:32933
I20260927 10:54:54.412818 5098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:32933 every 8 connection(s)
I20260927 10:54:54.413743 4970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:54.422819 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 4970
I20260927 10:54:54.422966 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:54.423262 1211 external_mini_cluster.cc:1428] Setting key aa778fa93198a83508c4ad3522da121e
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:54.428589 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.415569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:49241 (local address 127.1.46.254:45067)
0927 10:54:54.415728 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:54.415732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:54.416549 (+ 817us) server_negotiation.cc:407] Connection header received
0927 10:54:54.417924 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:54.417927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:54.417982 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:54.418083 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:54.419330 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.419859 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.420967 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.421193 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.421303 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.423990 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:54.423996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:54.424000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:54.424029 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:54.425893 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.426557 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.426560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.426562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.426691 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.427158 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.427162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.427165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.427359 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:54.427445 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:54.428198 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 10:54:54.428384 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:54:54.429747 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.415939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45067 (local address 127.1.46.194:49241)
0927 10:54:54.416386 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:54.416427 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:54.417715 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:54.418274 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:54.418281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:54.418726 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:54.419185 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:54.419197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.419980 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.419986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:54.420844 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:54.420853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.421334 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.421337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:54.421498 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.422144 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:54.422159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:54.423816 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:54.426012 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:54.426020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:54.426072 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:54.426447 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:54.426811 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:54.426817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:54.426820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:54.426993 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:54.427534 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:54.427540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:54.427840 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 10:54:54.428766 (+ 926us) client_negotiation.cc:240] Negotiation successful
0927 10:54:54.429054 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260927 10:54:54.431000 5099 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45067
I20260927 10:54:54.431214 5099 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:54.431672 5099 heartbeater.cc:511] Master 127.1.46.254:45067 requested a full tablet report, sending...
I20260927 10:54:54.432717 4774 ts_manager.cc:194] Registered new tserver with Master: 9b9ed40b3ec945839b990b68ff0cad1e (127.1.46.194:32933)
I20260927 10:54:54.433293 4774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:49241
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:54.451272 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45067
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:54.563364 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:54.563603 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:54.563644 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:54.567950 5104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:54.568005 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:54.568063 5104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:54.573204 5104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:54.574177 5104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:54.575279 5104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:54.584868 5109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:54.584868 5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:54.584901 5112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:54.585165 5104 server_base.cc:1034] running on GCE node
I20260927 10:54:54.585513 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:54.586156 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:54.587334 5104 hybrid_clock.cc:648] HybridClock initialized: now 1790506494587320 us; error 24 us; skew 500 ppm
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:54.590250 5104 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:54.591213 5104 webserver.cc:460] Webserver started at http://127.1.46.195:39721/ using document root <none> and password file <none>
I20260927 10:54:54.591811 5104 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:54.591908 5104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:54.592134 5104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:54.594100 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "ed12308cc1284402a1b2b9611d66a12d"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "edb4b6c6ad9a7c6f2e4dced759312be7"
server_key_iv: "302b5749d3936c12502105cb11b2aac3"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.594719 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "ed12308cc1284402a1b2b9611d66a12d"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "edb4b6c6ad9a7c6f2e4dced759312be7"
server_key_iv: "302b5749d3936c12502105cb11b2aac3"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.598371 5104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:54.600822 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.602036 5104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:54:54.602169 5104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "ed12308cc1284402a1b2b9611d66a12d"
format_stamp: "Formatted at 2026-09-27 10:54:54 on dist-test-slave-cffh"
server_key: "edb4b6c6ad9a7c6f2e4dced759312be7"
server_key_iv: "302b5749d3936c12502105cb11b2aac3"
server_key_version: "encryptionkey@0"
I20260927 10:54:54.602276 5104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:54.621030 5104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:54.624984 5104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:54.625195 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:54.625973 5104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:54.626930 5104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:54.626979 5104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.627019 5104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:54.627034 5104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:54.636130 5104 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:42733
I20260927 10:54:54.636157 5232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:42733 every 8 connection(s)
I20260927 10:54:54.637004 5104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:54.647033 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 5104
I20260927 10:54:54.647212 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:54.647606 1211 external_mini_cluster.cc:1428] Setting key c79e9cec87b056450467e4fd731b01cd
I20260927 10:54:54.650816 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.638842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:48251 (local address 127.1.46.254:45067)
0927 10:54:54.638999 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:54.639004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:54.639817 (+ 813us) server_negotiation.cc:407] Connection header received
0927 10:54:54.641278 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:54.641281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:54.641326 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:54.641408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:54.642531 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.643127 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.644321 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.644519 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.644590 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.647172 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:54.647186 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:54.647192 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:54.647223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:54.648762 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.649372 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.649375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.649376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.649448 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.649840 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.649843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.649844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.650012 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:54.650087 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:54.650538 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 10:54:54.650696 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:54:54.651662 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.639224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45067 (local address 127.1.46.195:48251)
0927 10:54:54.639662 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:54.639711 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:54.640996 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:54.641564 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:54.641570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:54.641913 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:54.642406 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:54.642419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.643254 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.643257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:54.644195 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:54.644205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.644654 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.644657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:54.644794 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.645335 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:54.645352 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:54.646955 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:54.648855 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:54.648864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:54.648881 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:54.649260 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:54.649550 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:54.649554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:54.649556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:54.649741 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:54.650108 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:54.650113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:54.650333 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 10:54:54.650498 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 10:54:54.650824 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260927 10:54:54.652762 5233 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45067
I20260927 10:54:54.652976 5233 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:54.653426 5233 heartbeater.cc:511] Master 127.1.46.254:45067 requested a full tablet report, sending...
I20260927 10:54:54.654388 4774 ts_manager.cc:194] Registered new tserver with Master: ed12308cc1284402a1b2b9611d66a12d (127.1.46.195:42733)
I20260927 10:54:54.654989 4774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:48251
I20260927 10:54:54.661514 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:54:54.687956 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.678138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.1.46.193:46509)
0927 10:54:54.678487 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:54.678503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:54.678532 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:54:54.678812 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:54.678818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:54.679016 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:54.679152 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:54.679808 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.680650 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.681491 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.681867 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.681958 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.684406 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:54.684414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:54.684426 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:54.684449 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:54.686227 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.686720 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.686727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.686728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.686810 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.687057 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.687060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.687061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.687296 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:54.687459 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:54.687678 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 10:54:54.687806 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":133,"threads_started":1}
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[4729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:54.699097 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:54.691124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.1.46.254:45067)
0927 10:54:54.691297 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:54.691303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:54.691321 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:54:54.691478 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:54.691482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:54.691533 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:54.691616 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:54.692207 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.692767 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.693546 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:54.693780 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:54.693886 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:54.696179 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:54.696183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:54.696193 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:54.696217 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:54.697402 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.697813 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.697817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.697820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.697900 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:54.698172 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:54.698177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:54.698180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:54.698393 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:54.698536 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:54.698813 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 10:54:54.698932 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:54:54.699486 4774 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51472
I20260927 10:54:54.700109 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4836
I20260927 10:54:54.705719 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4970
I20260927 10:54:54.711107 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 5104
I20260927 10:54:54.716156 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 4742
2026-09-27T10:54:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2129 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[5245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[5245](info): set up 2 sockets
Sep 27 10:54:54 dist-test-slave-cffh krb5kdc[5245](info): commencing operation
krb5kdc: starting...
W20260927 10:54:55.746876 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:54:55 dist-test-slave-cffh krb5kdc[5245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:54:55.865715 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45281
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:55.973853 5258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:55.974098 5258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:55.974146 5258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:55.978060 5258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:55.978109 5258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:55.978128 5258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:55.978152 5258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:55.978168 5258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:55.983877 5258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:55.984807 5258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:55.985752 5258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:55.993774 5263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:55.993821 5266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:55.993867 5258 server_base.cc:1034] running on GCE node
W20260927 10:54:55.993774 5264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:55.994670 5258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:55.995747 5258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:55.996902 5258 hybrid_clock.cc:648] HybridClock initialized: now 1790506495996871 us; error 47 us; skew 500 ppm
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:56.006798 5258 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:54:56.007525 5258 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:56.007612 5258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:56.007802 5258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:56.009894 5258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "1923cf8d3f4f42fb932f7490743da2f9"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "1d7fe7aea7904d35e86e1a6fccace1f8"
server_key_iv: "05e5b9fdb23ea436a2be86529d857a6b"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.010430 5258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "1923cf8d3f4f42fb932f7490743da2f9"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "1d7fe7aea7904d35e86e1a6fccace1f8"
server_key_iv: "05e5b9fdb23ea436a2be86529d857a6b"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.014287 5258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:54:56.016781 5272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.018476 5258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:56.018633 5258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "1923cf8d3f4f42fb932f7490743da2f9"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "1d7fe7aea7904d35e86e1a6fccace1f8"
server_key_iv: "05e5b9fdb23ea436a2be86529d857a6b"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.018740 5258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:56.036478 5258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:56.039398 5258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:56.039587 5258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:56.046118 5258 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45281
I20260927 10:54:56.046370 5324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45281 every 8 connection(s)
I20260927 10:54:56.046825 5258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:56.049461 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:56.051700 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 5258
I20260927 10:54:56.051793 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:56.052068 1211 external_mini_cluster.cc:1428] Setting key 3755cd848dba671fc2443045e686cbd2
I20260927 10:54:56.055106 5325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9: Bootstrap starting.
I20260927 10:54:56.057502 5325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:56.058292 5325 log.cc:824] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:56.060559 5325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9: No bootstrap required, opened a new log
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:56.063870 5325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1923cf8d3f4f42fb932f7490743da2f9" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45281 } }
I20260927 10:54:56.064108 5325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:56.064198 5325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1923cf8d3f4f42fb932f7490743da2f9, State: Initialized, Role: FOLLOWER
I20260927 10:54:56.064666 5325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1923cf8d3f4f42fb932f7490743da2f9" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45281 } }
I20260927 10:54:56.064815 5325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:56.064929 5325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:56.065048 5325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:56.065901 5325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1923cf8d3f4f42fb932f7490743da2f9" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45281 } }
I20260927 10:54:56.066238 5325 leader_election.cc:302] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1923cf8d3f4f42fb932f7490743da2f9; no voters:
I20260927 10:54:56.066536 5325 leader_election.cc:288] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:56.066653 5330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:56.066922 5330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [term 1 LEADER]: Becoming Leader. State: Replica: 1923cf8d3f4f42fb932f7490743da2f9, State: Running, Role: LEADER
I20260927 10:54:56.067076 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.053343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.1.46.254:45281)
0927 10:54:56.053716 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.053730 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.053763 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:54:56.054372 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.054388 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.054781 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.055045 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:56.055894 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.056858 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.058011 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.058560 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.058786 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.061496 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:56.061501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:56.061513 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:56.061538 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:56.063773 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.064253 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.064259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.064265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.064425 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.064707 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.064711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.064714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.065288 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:56.065490 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.065795 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.066093 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":121,"threads_started":1}
I20260927 10:54:56.067279 5330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1923cf8d3f4f42fb932f7490743da2f9" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45281 } }
I20260927 10:54:56.067836 5325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:56.068394 5332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1923cf8d3f4f42fb932f7490743da2f9. Latest consensus state: current_term: 1 leader_uuid: "1923cf8d3f4f42fb932f7490743da2f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1923cf8d3f4f42fb932f7490743da2f9" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45281 } } }
I20260927 10:54:56.068605 5332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:56.068841 5331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1923cf8d3f4f42fb932f7490743da2f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1923cf8d3f4f42fb932f7490743da2f9" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45281 } } }
I20260927 10:54:56.068958 5331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:56.069242 5338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:56.072383 5338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:56.078302 5338 catalog_manager.cc:1347] Generated new cluster ID: ecce26a11a11498d81538511dc603cbb
I20260927 10:54:56.078374 5338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:56.106032 5338 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:56.107053 5338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:56.112754 5338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9: Generated new TSK 0
I20260927 10:54:56.113416 5338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:54:56.141642 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45281
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:56.271534 5351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:56.271793 5351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:56.271879 5351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:56.275947 5351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:56.276022 5351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:56.276137 5351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:56.281406 5351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:56.282387 5351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:56.283406 5351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:56.292814 5356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:56.292975 5351 server_base.cc:1034] running on GCE node
W20260927 10:54:56.292814 5359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.293140 5357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:56.293521 5351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:56.294050 5351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:56.295218 5351 hybrid_clock.cc:648] HybridClock initialized: now 1790506496295204 us; error 28 us; skew 500 ppm
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:56.297797 5351 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:54:56.298472 5351 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:56.298563 5351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:56.298823 5351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:56.301020 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "83442481924643e1b5ce182698b6e504"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "1e2251662d1cf3c1825a2d533d74558b"
server_key_iv: "fe628bc48c6506b01335006e50cbe3ff"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.301699 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "83442481924643e1b5ce182698b6e504"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "1e2251662d1cf3c1825a2d533d74558b"
server_key_iv: "fe628bc48c6506b01335006e50cbe3ff"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.305612 5351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:54:56.307902 5365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.309275 5351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:56.309377 5351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "83442481924643e1b5ce182698b6e504"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "1e2251662d1cf3c1825a2d533d74558b"
server_key_iv: "fe628bc48c6506b01335006e50cbe3ff"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.309468 5351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:56.327693 5351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:56.330621 5351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:56.330825 5351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:56.331405 5351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:56.332196 5351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:56.332242 5351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.332302 5351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:56.332337 5351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.341434 5351 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:34065
I20260927 10:54:56.341447 5478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:34065 every 8 connection(s)
I20260927 10:54:56.342039 5351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:56.348347 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 5351
I20260927 10:54:56.348487 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:56.348788 1211 external_mini_cluster.cc:1428] Setting key 34087b4c0736d9eba8700779175e7fa1
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:56.356580 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.343757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:47509 (local address 127.1.46.254:45281)
0927 10:54:56.343904 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.343907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.344688 (+ 781us) server_negotiation.cc:407] Connection header received
0927 10:54:56.345985 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.345988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.346041 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.346154 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:56.347356 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.347886 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.349155 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.349386 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.349498 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.352141 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:56.352144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:56.352146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:56.352167 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:56.353629 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.354290 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.354296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.354300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.354427 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.354941 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.354947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.354950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.355227 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:56.355339 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.356065 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.356295 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:54:56.357409 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.344120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45281 (local address 127.1.46.193:47509)
0927 10:54:56.344532 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.344572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.345788 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.346341 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.346347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.346734 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:56.347213 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.347227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.348013 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.348019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.349036 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.349045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.349531 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.349535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.349700 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.350286 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:56.350301 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:56.351954 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:56.353745 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.353754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.353771 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.354143 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.354519 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.354524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.354526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.354777 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.355363 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:56.355371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:56.355721 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.355950 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.356330 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:54:56.359637 5479 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45281
I20260927 10:54:56.359901 5479 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:56.360428 5479 heartbeater.cc:511] Master 127.1.46.254:45281 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:54:56.362402 5289 ts_manager.cc:194] Registered new tserver with Master: 83442481924643e1b5ce182698b6e504 (127.1.46.193:34065)
I20260927 10:54:56.364113 5289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:47509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:54:56.377684 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45281
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:56.485291 5484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:56.485574 5484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:56.485674 5484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:56.489648 5484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:56.489760 5484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:56.489879 5484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:56.494982 5484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:56.495981 5484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:56.496959 5484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:56.506081 5490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.506093 5492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.506099 5489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:56.506490 5484 server_base.cc:1034] running on GCE node
I20260927 10:54:56.506950 5484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:56.507511 5484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:56.508704 5484 hybrid_clock.cc:648] HybridClock initialized: now 1790506496508681 us; error 34 us; skew 500 ppm
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:56.511390 5484 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:54:56.512104 5484 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:56.512212 5484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:56.512398 5484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:56.514294 5484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "9ead1b9664e2458ea6b18e2f39dbe74c"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "32bf52bb372a6a0e2ea71415905fd0fd"
server_key_iv: "57dc311ec9b9d796d3ea832341fce4a9"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.514839 5484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "9ead1b9664e2458ea6b18e2f39dbe74c"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "32bf52bb372a6a0e2ea71415905fd0fd"
server_key_iv: "57dc311ec9b9d796d3ea832341fce4a9"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.518229 5484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:54:56.520538 5498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.521692 5484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:54:56.521777 5484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "9ead1b9664e2458ea6b18e2f39dbe74c"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "32bf52bb372a6a0e2ea71415905fd0fd"
server_key_iv: "57dc311ec9b9d796d3ea832341fce4a9"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.521903 5484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:56.556546 5484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:56.559561 5484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:56.559747 5484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:56.560314 5484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:56.561120 5484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:56.561172 5484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.561285 5484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:56.561322 5484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.570650 5484 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:34315
I20260927 10:54:56.570688 5611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:34315 every 8 connection(s)
I20260927 10:54:56.571228 5484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:56.573196 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 5484
I20260927 10:54:56.573307 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:56.573572 1211 external_mini_cluster.cc:1428] Setting key 189578911d004024048d3e3fba75fad7
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
I20260927 10:54:56.586486 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.572946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:48295 (local address 127.1.46.254:45281)
0927 10:54:56.573096 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.573103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.573992 (+ 889us) server_negotiation.cc:407] Connection header received
0927 10:54:56.575499 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.575502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.575559 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.575652 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:56.576765 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.577298 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.578484 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.578892 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.579058 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.582229 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:56.582233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:56.582235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:56.582267 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:56.584114 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.584655 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.584660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.584662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.584764 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.585083 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.585086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.585087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.585294 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:56.585370 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.585990 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.586204 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:54:56.587343 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.573316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45281 (local address 127.1.46.194:48295)
0927 10:54:56.573822 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.573883 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.575295 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.575802 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.575809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.576165 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:56.576616 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.576628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.577424 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.577429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.578283 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.578291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.579124 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.579129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.579372 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.580157 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:56.580173 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:56.582055 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:56.584270 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.584275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.584286 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.584532 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.584864 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.584867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.584868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.584993 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.585388 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:56.585393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:56.585722 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.585916 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.586289 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260927 10:54:56.588869 5612 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45281
I20260927 10:54:56.589123 5612 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:56.589521 5612 heartbeater.cc:511] Master 127.1.46.254:45281 requested a full tablet report, sending...
I20260927 10:54:56.590507 5289 ts_manager.cc:194] Registered new tserver with Master: 9ead1b9664e2458ea6b18e2f39dbe74c (127.1.46.194:34315)
I20260927 10:54:56.591225 5289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:48295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:54:56.602715 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45281
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:54:56.708818 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:56.709041 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:56.709084 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:56.713157 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:56.713212 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:56.713331 5617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:56.718498 5617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:56.719477 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:56.720427 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:56.729509 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.729516 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:56.729535 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:56.729755 5617 server_base.cc:1034] running on GCE node
I20260927 10:54:56.730204 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:56.730917 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:56.732096 5617 hybrid_clock.cc:648] HybridClock initialized: now 1790506496732070 us; error 44 us; skew 500 ppm
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:54:56.734843 5617 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:54:56.735524 5617 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:56.735612 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:56.735836 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:56.737841 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "8e886430927f414d9fc3507ce0b2fd51"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "08cf53fe615f1fb2f88ea35c3e9b93d6"
server_key_iv: "b63c56488095afe5fe995ffea180dd94"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.738384 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "8e886430927f414d9fc3507ce0b2fd51"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "08cf53fe615f1fb2f88ea35c3e9b93d6"
server_key_iv: "b63c56488095afe5fe995ffea180dd94"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.741909 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:54:56.744324 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.745443 5617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:54:56.745576 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "8e886430927f414d9fc3507ce0b2fd51"
format_stamp: "Formatted at 2026-09-27 10:54:56 on dist-test-slave-cffh"
server_key: "08cf53fe615f1fb2f88ea35c3e9b93d6"
server_key_iv: "b63c56488095afe5fe995ffea180dd94"
server_key_version: "encryptionkey@0"
I20260927 10:54:56.745682 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:56.758867 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:56.761552 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:56.761746 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:56.762339 5617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:56.763164 5617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:56.763252 5617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.763331 5617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:56.763377 5617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:56.772770 5617 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:35119
I20260927 10:54:56.772792 5744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:35119 every 8 connection(s)
I20260927 10:54:56.773298 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:56.778152 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 5617
I20260927 10:54:56.778303 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:56.778630 1211 external_mini_cluster.cc:1428] Setting key 22e579d44b753598d2a4897614b1b9fc
Sep 27 10:54:56 dist-test-slave-cffh krb5kdc[5245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:54:56.786186 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.774942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:48867 (local address 127.1.46.254:45281)
0927 10:54:56.775102 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.775108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.775965 (+ 857us) server_negotiation.cc:407] Connection header received
0927 10:54:56.777191 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.777194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.777244 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.777328 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:56.778434 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.779192 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.780645 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.780866 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.780975 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.783338 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:56.783341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:56.783343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:56.783364 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:56.784541 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.784969 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.784972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.784973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.785039 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.785332 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.785334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.785335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.785490 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:56.785581 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.785976 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.786067 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:54:56.786993 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.775353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45281 (local address 127.1.46.195:48867)
0927 10:54:56.775811 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.775851 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.776993 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.777478 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.777484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.777827 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:56.778295 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.778314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.779327 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.779332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.780536 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.780546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.781006 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.781009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.781183 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.781750 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:54:56.781765 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:56.783197 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:56.784643 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.784647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.784657 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.784888 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.785126 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:54:56.785128 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:54:56.785132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:54:56.785256 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:54:56.785586 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:56.785590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:56.785789 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.785931 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.786181 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":132,"threads_started":1}
I20260927 10:54:56.788137 5745 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45281
I20260927 10:54:56.788384 5745 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:56.788806 5745 heartbeater.cc:511] Master 127.1.46.254:45281 requested a full tablet report, sending...
I20260927 10:54:56.789762 5289 ts_manager.cc:194] Registered new tserver with Master: 8e886430927f414d9fc3507ce0b2fd51 (127.1.46.195:35119)
I20260927 10:54:56.790300 5289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:48867
I20260927 10:54:56.792160 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:54:56.800964 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.793962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.1.46.254:45281)
0927 10:54:56.794095 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.794099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.794305 (+ 206us) server_negotiation.cc:407] Connection header received
0927 10:54:56.794383 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.794385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.794419 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.794478 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:56.795207 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.795674 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.796555 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.796752 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.796823 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.798068 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:56.798071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:56.798073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:54:56.798092 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:56.799489 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.799879 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.799882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.799883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.799957 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:54:56.800238 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:54:56.800240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:54:56.800241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:54:56.800419 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:56.800514 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.800711 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.800791 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:54:56.803625 5289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:56.806495 5289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:56.821110 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.815721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.1.46.193:34065)
0927 10:54:56.816022 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.816031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.816286 (+ 255us) server_negotiation.cc:407] Connection header received
0927 10:54:56.816402 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.816407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.816603 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.816761 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:56.817270 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.818117 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.819702 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.820349 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.820487 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.820641 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.820841 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.820933 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":102,"threads_started":1}
I20260927 10:54:56.821100 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.815543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:34065 (local address 127.0.0.1:55316)
0927 10:54:56.816170 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.816184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.816283 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.816956 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.816960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.816979 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:56.817163 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.817172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.818288 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.818291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.819573 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.819584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.820485 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.820489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.820610 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.820665 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.820811 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.820934 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":43,"thread_start_us":85,"threads_started":1}
I20260927 10:54:56.821419 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.815105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.1.46.194:34315)
0927 10:54:56.816318 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.816331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.816350 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:54:56.816410 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.816414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.816603 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.816761 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:56.817096 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.817903 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.819778 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.820569 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.820674 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.820800 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.821175 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.821269 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":127,"threads_started":1}
I20260927 10:54:56.821978 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.814821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:35119 (local address 127.0.0.1:59390)
0927 10:54:56.815342 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.815395 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.815560 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.816976 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.816979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.816990 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:56.817163 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.817170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.819331 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.819334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.820514 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.820524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.821569 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.821572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.821683 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.821699 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.821739 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.821792 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":43,"thread_start_us":88,"threads_started":1}
I20260927 10:54:56.822317 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.814948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:34315 (local address 127.0.0.1:55470)
0927 10:54:56.815503 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.815517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.815596 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.816772 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.816778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.816802 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:56.816963 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.816972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.818034 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.818037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.819625 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.819662 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.820856 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.820860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.820944 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.820966 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.821032 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.821095 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":27,"thread_start_us":78,"threads_started":1}
I20260927 10:54:56.822490 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.815831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.1.46.195:35119)
0927 10:54:56.816527 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.816543 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.816574 (+ 31us) server_negotiation.cc:407] Connection header received
0927 10:54:56.816639 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.816643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.816842 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.817007 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:56.818041 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.819175 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.820682 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.821428 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.822024 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.822162 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.822234 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.822316 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":155,"threads_started":1}
I20260927 10:54:56.824569 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 262153fc53f743c292a992f3cfbce094 (DEFAULT_TABLE table=test-table [id=60832da528fa4afe93f28aa56a4212a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:56.825621 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262153fc53f743c292a992f3cfbce094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:56.825304 5679 tablet_service.cc:1467] Processing CreateTablet for tablet 262153fc53f743c292a992f3cfbce094 (DEFAULT_TABLE table=test-table [id=60832da528fa4afe93f28aa56a4212a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:56.825317 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 262153fc53f743c292a992f3cfbce094 (DEFAULT_TABLE table=test-table [id=60832da528fa4afe93f28aa56a4212a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:56.826416 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262153fc53f743c292a992f3cfbce094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:56.826423 5679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262153fc53f743c292a992f3cfbce094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:56.832242 5762 tablet_bootstrap.cc:492] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504: Bootstrap starting.
I20260927 10:54:56.833895 5763 tablet_bootstrap.cc:492] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Bootstrap starting.
I20260927 10:54:56.834223 5762 tablet_bootstrap.cc:654] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:56.834906 5764 tablet_bootstrap.cc:492] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Bootstrap starting.
I20260927 10:54:56.835589 5762 log.cc:824] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:56.836203 5763 tablet_bootstrap.cc:654] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:56.837006 5764 tablet_bootstrap.cc:654] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:56.837433 5763 log.cc:824] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:56.837895 5762 tablet_bootstrap.cc:492] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504: No bootstrap required, opened a new log
I20260927 10:54:56.838092 5762 ts_tablet_manager.cc:1397] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260927 10:54:56.838375 5764 log.cc:824] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:56.839411 5763 tablet_bootstrap.cc:492] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: No bootstrap required, opened a new log
I20260927 10:54:56.839582 5763 ts_tablet_manager.cc:1397] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:54:56.840250 5764 tablet_bootstrap.cc:492] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: No bootstrap required, opened a new log
I20260927 10:54:56.840457 5764 ts_tablet_manager.cc:1397] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:54:56.841941 5762 raft_consensus.cc:348] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.842203 5762 raft_consensus.cc:374] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:56.842259 5762 raft_consensus.cc:729] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83442481924643e1b5ce182698b6e504, State: Initialized, Role: FOLLOWER
I20260927 10:54:56.842723 5762 consensus_queue.cc:260] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.843396 5479 heartbeater.cc:503] Master 127.1.46.254:45281 was elected leader, sending a full tablet report...
I20260927 10:54:56.843361 5763 raft_consensus.cc:348] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.843609 5763 raft_consensus.cc:374] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:56.843665 5763 raft_consensus.cc:729] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ead1b9664e2458ea6b18e2f39dbe74c, State: Initialized, Role: FOLLOWER
I20260927 10:54:56.843999 5764 raft_consensus.cc:348] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.844110 5763 consensus_queue.cc:260] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.844273 5764 raft_consensus.cc:374] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:56.844329 5764 raft_consensus.cc:729] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e886430927f414d9fc3507ce0b2fd51, State: Initialized, Role: FOLLOWER
I20260927 10:54:56.844790 5764 consensus_queue.cc:260] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.845057 5763 ts_tablet_manager.cc:1428] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:54:56.845377 5612 heartbeater.cc:503] Master 127.1.46.254:45281 was elected leader, sending a full tablet report...
I20260927 10:54:56.845474 5745 heartbeater.cc:503] Master 127.1.46.254:45281 was elected leader, sending a full tablet report...
I20260927 10:54:56.845819 5764 ts_tablet_manager.cc:1428] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:54:56.847517 5762 ts_tablet_manager.cc:1428] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 10:54:56.927143 5770 raft_consensus.cc:482] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:56.927371 5770 raft_consensus.cc:504] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.928524 5770 leader_election.cc:288] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ead1b9664e2458ea6b18e2f39dbe74c (127.1.46.194:34315), 83442481924643e1b5ce182698b6e504 (127.1.46.193:34065)
I20260927 10:54:56.933416 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.929326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:45283 (local address 127.1.46.193:34065)
0927 10:54:56.929476 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.929480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.929684 (+ 204us) server_negotiation.cc:407] Connection header received
0927 10:54:56.929823 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.929826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.929876 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.929952 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:56.930222 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.930876 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.932291 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.932874 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.932994 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.933027 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.933200 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.933294 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:54:56.933416 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.928965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:53057 (local address 127.1.46.194:34315)
0927 10:54:56.929114 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.929118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.929132 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:54:56.929285 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.929290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.929345 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.929434 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:54:56.929835 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.930616 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.932139 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.932753 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.932872 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.932943 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.933239 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.933310 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:54:56.933400 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.928863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:34315 (local address 127.1.46.195:53057)
0927 10:54:56.929028 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.929045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.929144 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.929531 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.929534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.929556 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:56.929707 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.929714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.930725 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.930727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.932007 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.932019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.932890 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.932893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.933032 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.933063 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.933216 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.933296 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:54:56.933411 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.929221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:34065 (local address 127.1.46.195:45283)
0927 10:54:56.929563 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:56.929576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:56.929699 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:56.929973 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:56.929975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:56.929990 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:54:56.930087 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.930092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.930973 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.930976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.932186 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:56.932194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.932982 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.932988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:56.933075 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.933109 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:54:56.933177 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 10:54:56.933254 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260927 10:54:56.934212 5566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262153fc53f743c292a992f3cfbce094" candidate_uuid: "8e886430927f414d9fc3507ce0b2fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" is_pre_election: true
I20260927 10:54:56.934512 5566 raft_consensus.cc:2454] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e886430927f414d9fc3507ce0b2fd51 in term 0.
I20260927 10:54:56.935089 5632 leader_election.cc:302] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e886430927f414d9fc3507ce0b2fd51, 9ead1b9664e2458ea6b18e2f39dbe74c; no voters:
I20260927 10:54:56.935269 5433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262153fc53f743c292a992f3cfbce094" candidate_uuid: "8e886430927f414d9fc3507ce0b2fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83442481924643e1b5ce182698b6e504" is_pre_election: true
I20260927 10:54:56.935395 5770 raft_consensus.cc:2781] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:56.935518 5770 raft_consensus.cc:482] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:56.935578 5770 raft_consensus.cc:3037] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:56.935556 5433 raft_consensus.cc:2454] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e886430927f414d9fc3507ce0b2fd51 in term 0.
I20260927 10:54:56.936758 5770 raft_consensus.cc:504] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.937079 5770 leader_election.cc:288] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 9ead1b9664e2458ea6b18e2f39dbe74c (127.1.46.194:34315), 83442481924643e1b5ce182698b6e504 (127.1.46.193:34065)
I20260927 10:54:56.937387 5566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262153fc53f743c292a992f3cfbce094" candidate_uuid: "8e886430927f414d9fc3507ce0b2fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c"
I20260927 10:54:56.937516 5433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262153fc53f743c292a992f3cfbce094" candidate_uuid: "8e886430927f414d9fc3507ce0b2fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83442481924643e1b5ce182698b6e504"
I20260927 10:54:56.937538 5566 raft_consensus.cc:3037] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:56.937628 5433 raft_consensus.cc:3037] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:56.938864 5566 raft_consensus.cc:2454] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e886430927f414d9fc3507ce0b2fd51 in term 1.
I20260927 10:54:56.938864 5433 raft_consensus.cc:2454] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e886430927f414d9fc3507ce0b2fd51 in term 1.
I20260927 10:54:56.939144 5632 leader_election.cc:302] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83442481924643e1b5ce182698b6e504, 8e886430927f414d9fc3507ce0b2fd51; no voters:
I20260927 10:54:56.939319 5770 raft_consensus.cc:2781] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:56.939540 5770 raft_consensus.cc:686] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 1 LEADER]: Becoming Leader. State: Replica: 8e886430927f414d9fc3507ce0b2fd51, State: Running, Role: LEADER
I20260927 10:54:56.939857 5770 consensus_queue.cc:237] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } }
I20260927 10:54:56.942975 5287 catalog_manager.cc:5495] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e886430927f414d9fc3507ce0b2fd51 (127.1.46.195). New cstate: current_term: 1 leader_uuid: "8e886430927f414d9fc3507ce0b2fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e886430927f414d9fc3507ce0b2fd51" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 35119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:56.972302 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:56.968561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.1.46.195:35119)
0927 10:54:56.968688 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:56.968692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:56.968771 (+ 79us) server_negotiation.cc:407] Connection header received
0927 10:54:56.968955 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:56.968957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:56.969010 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:56.969117 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:54:56.969378 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.969974 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.970850 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:56.971075 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:56.971178 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:56.971499 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:54:56.971889 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:54:56.972018 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:56.972085 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 10:54:56.972135 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:54:56.977742 5566 raft_consensus.cc:1261] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 1 FOLLOWER]: Refusing update from remote peer 8e886430927f414d9fc3507ce0b2fd51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:56.977743 5433 raft_consensus.cc:1261] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504 [term 1 FOLLOWER]: Refusing update from remote peer 8e886430927f414d9fc3507ce0b2fd51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:56.978405 5770 consensus_queue.cc:1035] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83442481924643e1b5ce182698b6e504" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:56.978574 5772 consensus_queue.cc:1035] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ead1b9664e2458ea6b18e2f39dbe74c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:56.987352 5777 mvcc.cc:205] Tried to move back new op lower bound from 7333914611614228480 to 7333914611468066816. Current Snapshot: MvccSnapshot[applied={T|T < 7333914611614228480}]
I20260927 10:54:56.993494 5287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:56.993642 5287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:56.995889 5287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1923cf8d3f4f42fb932f7490743da2f9: Sending DeleteTablet for 3 replicas of tablet 262153fc53f743c292a992f3cfbce094
I20260927 10:54:56.996591 5679 tablet_service.cc:1514] Processing DeleteTablet for tablet 262153fc53f743c292a992f3cfbce094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:56 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:59390
I20260927 10:54:56.996631 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 262153fc53f743c292a992f3cfbce094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:56 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:55470
I20260927 10:54:56.996644 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 262153fc53f743c292a992f3cfbce094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:56 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:55316
I20260927 10:54:56.997047 5786 tablet_replica.cc:330] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: stopping tablet replica
I20260927 10:54:56.997331 5786 raft_consensus.cc:2229] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:56.997386 5787 tablet_replica.cc:330] T 262153fc53f743c292a992f3cfbce094 P 83442481924643e1b5ce182698b6e504: stopping tablet replica
I20260927 10:54:56.997637 5786 raft_consensus.cc:2258] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:56.997694 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 5351
I20260927 10:54:56.998562 5785 tablet_replica.cc:330] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: stopping tablet replica
I20260927 10:54:56.998870 5785 raft_consensus.cc:2229] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:56.999133 5785 raft_consensus.cc:2258] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:57.000174 5786 ts_tablet_manager.cc:1905] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:57.000339 5785 ts_tablet_manager.cc:1905] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:57.002991 5786 ts_tablet_manager.cc:1918] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:57.002991 5785 ts_tablet_manager.cc:1918] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:57.003087 5786 log.cc:1196] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/262153fc53f743c292a992f3cfbce094
I20260927 10:54:57.003087 5785 log.cc:1196] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/262153fc53f743c292a992f3cfbce094
I20260927 10:54:57.003420 5786 ts_tablet_manager.cc:1939] T 262153fc53f743c292a992f3cfbce094 P 8e886430927f414d9fc3507ce0b2fd51: Deleting consensus metadata
I20260927 10:54:57.003420 5785 ts_tablet_manager.cc:1939] T 262153fc53f743c292a992f3cfbce094 P 9ead1b9664e2458ea6b18e2f39dbe74c: Deleting consensus metadata
I20260927 10:54:57.004683 5273 catalog_manager.cc:4841] TS 9ead1b9664e2458ea6b18e2f39dbe74c (127.1.46.194:34315): tablet 262153fc53f743c292a992f3cfbce094 (table test-table [id=60832da528fa4afe93f28aa56a4212a7]) successfully deleted
I20260927 10:54:57.004896 5275 catalog_manager.cc:4841] TS 8e886430927f414d9fc3507ce0b2fd51 (127.1.46.195:35119): tablet 262153fc53f743c292a992f3cfbce094 (table test-table [id=60832da528fa4afe93f28aa56a4212a7]) successfully deleted
W20260927 10:54:57.008095 5273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:34065 (error 108)
I20260927 10:54:57.008242 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 5484
I20260927 10:54:57.010383 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.010015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:34065 (local address 127.0.0.1:55324)
0927 10:54:57.010200 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.010281 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:34065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:54:57.010558 5273 catalog_manager.cc:4584] TS 83442481924643e1b5ce182698b6e504 (127.1.46.193:34065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 262153fc53f743c292a992f3cfbce094: Network error: Client connection negotiation failed: client connection to 127.1.46.193:34065: connect: Connection refused (error 111)
I20260927 10:54:57.014233 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 5617
I20260927 10:54:57.019891 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 5258
2026-09-27T10:54:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2303 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T10:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:57Z Disabled control of system clock
I20260927 10:54:57.049480 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36305
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:54:57.157642 5792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:57.157889 5792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:57.157925 5792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:57.161837 5792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:57.161893 5792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:57.161909 5792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:57.161926 5792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:57.161940 5792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:54:57.166621 5792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:36305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:57.167718 5792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:57.168998 5792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:57.177189 5800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:57.177320 5792 server_base.cc:1034] running on GCE node
W20260927 10:54:57.177271 5797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.177275 5798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:57.178047 5792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:57.179175 5792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:57.180356 5792 hybrid_clock.cc:648] HybridClock initialized: now 1790506497180344 us; error 27 us; skew 500 ppm
I20260927 10:54:57.182077 5792 webserver.cc:460] Webserver started at http://127.1.46.254:34033/ using document root <none> and password file <none>
I20260927 10:54:57.182737 5792 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:57.182828 5792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:57.183058 5792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:57.185141 5792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "216c1585bcfa4760946a5f2ef575ed4f"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "ca6e19e8c70a29023ef0c7d027c1b60d"
server_key_iv: "759ccb0d2d5bf956a7345b4be5a32fea"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.185712 5792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "216c1585bcfa4760946a5f2ef575ed4f"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "ca6e19e8c70a29023ef0c7d027c1b60d"
server_key_iv: "759ccb0d2d5bf956a7345b4be5a32fea"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.189284 5792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:54:57.191669 5806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.192914 5792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:57.193128 5792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "216c1585bcfa4760946a5f2ef575ed4f"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "ca6e19e8c70a29023ef0c7d027c1b60d"
server_key_iv: "759ccb0d2d5bf956a7345b4be5a32fea"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.193244 5792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:57.197896 5792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:57.198457 5792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:57.198674 5792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:57.204999 5792 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:36305
I20260927 10:54:57.205049 5858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:36305 every 8 connection(s)
I20260927 10:54:57.205885 5792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:57.208899 5859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:57.214078 5859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f: Bootstrap starting.
I20260927 10:54:57.214509 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 5792
I20260927 10:54:57.214669 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:57.214922 1211 external_mini_cluster.cc:1428] Setting key e04433c2ed20032814daedfa0deb9c27
I20260927 10:54:57.216631 5859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:57.217335 5859 log.cc:824] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:57.219287 5859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f: No bootstrap required, opened a new log
I20260927 10:54:57.220278 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.216120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.1.46.254:36305)
0927 10:54:57.216785 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.216801 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.216832 (+ 31us) server_negotiation.cc:407] Connection header received
0927 10:54:57.217457 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.217484 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.217903 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.218268 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.219016 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.219022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.219037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.219055 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.219103 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.219118 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.219241 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.219353 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.219589 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260927 10:54:57.222718 5859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216c1585bcfa4760946a5f2ef575ed4f" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36305 } }
I20260927 10:54:57.222939 5859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:57.223016 5859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216c1585bcfa4760946a5f2ef575ed4f, State: Initialized, Role: FOLLOWER
I20260927 10:54:57.223428 5859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216c1585bcfa4760946a5f2ef575ed4f" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36305 } }
I20260927 10:54:57.223577 5859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:57.223661 5859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:57.223783 5859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:57.224597 5859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216c1585bcfa4760946a5f2ef575ed4f" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36305 } }
I20260927 10:54:57.224924 5859 leader_election.cc:302] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 216c1585bcfa4760946a5f2ef575ed4f; no voters:
I20260927 10:54:57.225209 5859 leader_election.cc:288] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:57.225335 5864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:57.225545 5864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [term 1 LEADER]: Becoming Leader. State: Replica: 216c1585bcfa4760946a5f2ef575ed4f, State: Running, Role: LEADER
I20260927 10:54:57.225827 5864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216c1585bcfa4760946a5f2ef575ed4f" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36305 } }
I20260927 10:54:57.226485 5859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:57.227591 5865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "216c1585bcfa4760946a5f2ef575ed4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216c1585bcfa4760946a5f2ef575ed4f" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36305 } } }
I20260927 10:54:57.227595 5866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 216c1585bcfa4760946a5f2ef575ed4f. Latest consensus state: current_term: 1 leader_uuid: "216c1585bcfa4760946a5f2ef575ed4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216c1585bcfa4760946a5f2ef575ed4f" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36305 } } }
I20260927 10:54:57.227710 5865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:57.227721 5866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:57.228071 5873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:57.231002 5873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:57.235747 5873 catalog_manager.cc:1347] Generated new cluster ID: 562919980e3642bca61018b407159cec
I20260927 10:54:57.235802 5873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:57.252795 5873 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:57.253122 5873 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:57.264494 5873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f: Generated new TSK 0
I20260927 10:54:57.265264 5873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:57.272289 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36305
--builtin_ntp_servers=127.1.46.212:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:54:57.382081 5883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:57.382365 5883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:57.382458 5883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:57.386431 5883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:57.386507 5883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:57.386610 5883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:54:57.391273 5883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:57.392277 5883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:57.393571 5883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:54:57.402781 5883 server_base.cc:1034] running on GCE node
W20260927 10:54:57.402737 5888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.402746 5891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.402761 5889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:57.403470 5883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:57.404066 5883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:57.405256 5883 hybrid_clock.cc:648] HybridClock initialized: now 1790506497405221 us; error 46 us; skew 500 ppm
I20260927 10:54:57.407061 5883 webserver.cc:460] Webserver started at http://127.1.46.193:40229/ using document root <none> and password file <none>
I20260927 10:54:57.407662 5883 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:57.407755 5883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:57.407974 5883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:57.409881 5883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "ced6413f719e4e63b74d28de0924fec5"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "60ace4f15ec6fa87507b6000ab29fa00"
server_key_iv: "6f1e2f4cf21120478a276821058a7084"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.410419 5883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "ced6413f719e4e63b74d28de0924fec5"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "60ace4f15ec6fa87507b6000ab29fa00"
server_key_iv: "6f1e2f4cf21120478a276821058a7084"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.413784 5883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 10:54:57.416100 5897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.417306 5883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:57.417430 5883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "ced6413f719e4e63b74d28de0924fec5"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "60ace4f15ec6fa87507b6000ab29fa00"
server_key_iv: "6f1e2f4cf21120478a276821058a7084"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.417529 5883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:57.421936 5883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:57.422456 5883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:57.422689 5883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:57.423195 5883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:57.424019 5883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:57.424088 5883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.424158 5883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:57.424213 5883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.433313 5883 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:42085
I20260927 10:54:57.433377 6010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:42085 every 8 connection(s)
I20260927 10:54:57.434183 5883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:54:57.438117 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 5883
I20260927 10:54:57.438269 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:54:57.438624 1211 external_mini_cluster.cc:1428] Setting key 4a86cedb74ecd0ad7a514a2a8103d02a
I20260927 10:54:57.440606 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.435991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:60875 (local address 127.1.46.254:36305)
0927 10:54:57.436134 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.436138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.437125 (+ 987us) server_negotiation.cc:407] Connection header received
0927 10:54:57.438682 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.438688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.438735 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.438832 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.440117 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.440122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.440128 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.440142 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.440164 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.440175 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.440270 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.440477 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.440510 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:54:57.440954 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36305
--builtin_ntp_servers=127.1.46.212:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:54:57.441753 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.436369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36305 (local address 127.1.46.193:60875)
0927 10:54:57.436983 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.437021 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.438423 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.439029 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.439040 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.439136 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.439815 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.439817 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.439855 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.439970 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.440309 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.440320 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.440327 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.440524 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.440785 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260927 10:54:57.442938 6011 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36305
I20260927 10:54:57.443169 6011 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:57.443558 6011 heartbeater.cc:511] Master 127.1.46.254:36305 requested a full tablet report, sending...
I20260927 10:54:57.445084 5823 ts_manager.cc:194] Registered new tserver with Master: ced6413f719e4e63b74d28de0924fec5 (127.1.46.193:42085)
W20260927 10:54:57.547518 6014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:57.547747 6014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:57.547784 6014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:57.551616 6014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:57.551658 6014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:57.551709 6014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:54:57.556129 6014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:57.557013 6014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:57.558401 6014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:57.567731 6019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.567759 6022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.567699 6020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:57.568174 6014 server_base.cc:1034] running on GCE node
I20260927 10:54:57.568553 6014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:57.569093 6014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:57.570254 6014 hybrid_clock.cc:648] HybridClock initialized: now 1790506497570239 us; error 28 us; skew 500 ppm
I20260927 10:54:57.572161 6014 webserver.cc:460] Webserver started at http://127.1.46.194:43771/ using document root <none> and password file <none>
I20260927 10:54:57.572718 6014 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:57.572781 6014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:57.572995 6014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:57.575109 6014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "71554465ff941f142f51e66e132f7111"
server_key_iv: "6b939d59c7c7f1603b99f8efea23695a"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.575662 6014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "71554465ff941f142f51e66e132f7111"
server_key_iv: "6b939d59c7c7f1603b99f8efea23695a"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.579264 6014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:54:57.581596 6028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.582895 6014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:54:57.583020 6014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "71554465ff941f142f51e66e132f7111"
server_key_iv: "6b939d59c7c7f1603b99f8efea23695a"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.583130 6014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:57.587566 6014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:57.588119 6014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:57.588301 6014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:57.588814 6014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:57.589668 6014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:57.589741 6014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.589808 6014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:57.589864 6014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.598999 6014 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:35365
I20260927 10:54:57.599073 6141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:35365 every 8 connection(s)
I20260927 10:54:57.599884 6014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:54:57.605580 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.601585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:33021 (local address 127.1.46.254:36305)
0927 10:54:57.601743 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.601747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.602781 (+ 1034us) server_negotiation.cc:407] Connection header received
0927 10:54:57.604037 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.604039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.604071 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.604143 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.605195 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.605199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.605201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.605211 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.605244 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.605252 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.605324 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.605445 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.605480 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:54:57.606418 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.601984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36305 (local address 127.1.46.194:33021)
0927 10:54:57.602649 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.602688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.603843 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.604318 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.604324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.604408 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.604953 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.604954 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.604992 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.605055 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.605326 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.605332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.605336 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.605460 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.605665 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260927 10:54:57.607553 6142 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36305
I20260927 10:54:57.607807 6142 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:57.608043 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6014
I20260927 10:54:57.608157 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:54:57.608268 6142 heartbeater.cc:511] Master 127.1.46.254:36305 requested a full tablet report, sending...
I20260927 10:54:57.608465 1211 external_mini_cluster.cc:1428] Setting key 5b7f6e4fd5be353e057bcc4439055b3b
I20260927 10:54:57.609331 5823 ts_manager.cc:194] Registered new tserver with Master: c8b9f4efa8ed44a6b0b42a07f8e1d83c (127.1.46.194:35365)
I20260927 10:54:57.610641 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36305
--builtin_ntp_servers=127.1.46.212:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:54:57.716508 6145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:57.716809 6145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:57.716884 6145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:57.720989 6145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:57.721069 6145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:57.721163 6145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:54:57.725819 6145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:57.726964 6145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:57.728415 6145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:57.737860 6151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.737903 6150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:57.737874 6153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:57.738027 6145 server_base.cc:1034] running on GCE node
I20260927 10:54:57.738509 6145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:57.739089 6145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:57.740257 6145 hybrid_clock.cc:648] HybridClock initialized: now 1790506497740244 us; error 25 us; skew 500 ppm
I20260927 10:54:57.741958 6145 webserver.cc:460] Webserver started at http://127.1.46.195:46399/ using document root <none> and password file <none>
I20260927 10:54:57.742506 6145 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:57.742606 6145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:57.742830 6145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:57.744752 6145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "8c30058d14894cf28d6e366c16dc19bc"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "6d76fbb887669968aa77d49d12b9c06c"
server_key_iv: "af70eeb4ccc82c1116d8b48ef47922f8"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.745302 6145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "8c30058d14894cf28d6e366c16dc19bc"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "6d76fbb887669968aa77d49d12b9c06c"
server_key_iv: "af70eeb4ccc82c1116d8b48ef47922f8"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.748721 6145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 10:54:57.751003 6159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.752146 6145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:54:57.752280 6145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "8c30058d14894cf28d6e366c16dc19bc"
format_stamp: "Formatted at 2026-09-27 10:54:57 on dist-test-slave-cffh"
server_key: "6d76fbb887669968aa77d49d12b9c06c"
server_key_iv: "af70eeb4ccc82c1116d8b48ef47922f8"
server_key_version: "encryptionkey@0"
I20260927 10:54:57.752388 6145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:57.756790 6145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:57.757334 6145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:57.757505 6145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:57.758004 6145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:54:57.758895 6145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:54:57.758971 6145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.759059 6145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:54:57.759110 6145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:57.767836 6145 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:46369
I20260927 10:54:57.767861 6272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:46369 every 8 connection(s)
I20260927 10:54:57.768738 6145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:54:57.774072 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.770331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:42073 (local address 127.1.46.254:36305)
0927 10:54:57.770450 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.770454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.771483 (+ 1029us) server_negotiation.cc:407] Connection header received
0927 10:54:57.772664 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.772666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.772702 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.772776 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.773763 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.773765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.773767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.773774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.773789 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.773795 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.773859 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.773945 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.773977 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:54:57.774940 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.770758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36305 (local address 127.1.46.195:42073)
0927 10:54:57.771350 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.771389 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.772485 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.772890 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.772896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.772976 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.773550 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.773551 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.773587 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.773649 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.773847 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.773851 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.773854 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.773965 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.774203 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260927 10:54:57.775915 6273 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36305
I20260927 10:54:57.776137 6273 heartbeater.cc:464] Registering TS with master...
I20260927 10:54:57.776261 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6145
I20260927 10:54:57.776361 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:54:57.776562 6273 heartbeater.cc:511] Master 127.1.46.254:36305 requested a full tablet report, sending...
I20260927 10:54:57.776650 1211 external_mini_cluster.cc:1428] Setting key 475cd192ad4cb342805dfeb73893ea46
I20260927 10:54:57.777513 5823 ts_manager.cc:194] Registered new tserver with Master: 8c30058d14894cf28d6e366c16dc19bc (127.1.46.195:46369)
I20260927 10:54:57.778592 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:54:57.782047 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.780543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.1.46.254:36305)
0927 10:54:57.780662 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.780668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.780855 (+ 187us) server_negotiation.cc:407] Connection header received
0927 10:54:57.780981 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.780984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.781021 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.781102 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.781678 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.781683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.781686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.781697 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.781715 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.781724 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.781818 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.781903 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.781937 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:54:57.784428 5822 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:54:57.787426 5822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:54:57.798785 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.795252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:46369 (local address 127.0.0.1:44366)
0927 10:54:57.796256 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.796272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.796352 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.797226 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.797233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.797305 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.797893 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.797896 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.797917 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.797986 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.798350 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.798360 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.798366 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.798483 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.798552 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":80,"threads_started":1}
I20260927 10:54:57.799208 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.795198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:35365 (local address 127.0.0.1:46796)
0927 10:54:57.796070 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.796118 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.796219 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.797369 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.797374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.797396 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.798328 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.798332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.798345 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.798371 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.798751 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.798754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.798758 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.799056 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.799095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":70,"threads_started":1}
I20260927 10:54:57.799240 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.796054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.1.46.195:46369)
0927 10:54:57.796695 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.796708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.796735 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:54:57.796790 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.796797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.796951 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.797164 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.798157 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.798165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.798177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.798190 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.798232 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.798264 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.798375 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.798993 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.799073 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":94,"threads_started":1}
I20260927 10:54:57.799644 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.795299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.1.46.194:35365)
0927 10:54:57.795713 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.795728 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.796324 (+ 596us) server_negotiation.cc:407] Connection header received
0927 10:54:57.796382 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.796388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.796575 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.796704 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.798497 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.798502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.798510 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.798522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.798560 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.798612 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.798699 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.799104 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.799165 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260927 10:54:57.799698 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.795899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:42085 (local address 127.0.0.1:41102)
0927 10:54:57.796577 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.796591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.796667 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.797667 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.797671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.797689 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.798251 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.798253 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.798263 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.798289 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.798644 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.798648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.798653 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.799534 (+ 881us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.799572 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260927 10:54:57.800169 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.796041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.1.46.193:42085)
0927 10:54:57.796438 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.796449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.796753 (+ 304us) server_negotiation.cc:407] Connection header received
0927 10:54:57.796810 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.796815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.796978 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.797104 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.798418 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.798424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.798436 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.798449 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.798492 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.798521 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.798647 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.799904 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.799974 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":95,"threads_started":1}
I20260927 10:54:57.802110 6207 tablet_service.cc:1467] Processing CreateTablet for tablet 9353cbe32de74f20aff9037c00e2e8ee (DEFAULT_TABLE table=test-table [id=520ffedc6a2f41558e4a0936f95d8d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:57.802616 5945 tablet_service.cc:1467] Processing CreateTablet for tablet 9353cbe32de74f20aff9037c00e2e8ee (DEFAULT_TABLE table=test-table [id=520ffedc6a2f41558e4a0936f95d8d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:57.802731 6076 tablet_service.cc:1467] Processing CreateTablet for tablet 9353cbe32de74f20aff9037c00e2e8ee (DEFAULT_TABLE table=test-table [id=520ffedc6a2f41558e4a0936f95d8d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:54:57.803227 6207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9353cbe32de74f20aff9037c00e2e8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:57.803635 6076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9353cbe32de74f20aff9037c00e2e8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:57.803656 5945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9353cbe32de74f20aff9037c00e2e8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:57.809442 6290 tablet_bootstrap.cc:492] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5: Bootstrap starting.
I20260927 10:54:57.810458 6292 tablet_bootstrap.cc:492] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Bootstrap starting.
I20260927 10:54:57.810631 6291 tablet_bootstrap.cc:492] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Bootstrap starting.
I20260927 10:54:57.812177 6290 tablet_bootstrap.cc:654] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:57.812548 6291 tablet_bootstrap.cc:654] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:57.812547 6292 tablet_bootstrap.cc:654] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:57.812979 6290 log.cc:824] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:57.813364 6292 log.cc:824] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:57.813364 6291 log.cc:824] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:57.814899 6290 tablet_bootstrap.cc:492] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5: No bootstrap required, opened a new log
I20260927 10:54:57.815064 6291 tablet_bootstrap.cc:492] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: No bootstrap required, opened a new log
I20260927 10:54:57.815093 6290 ts_tablet_manager.cc:1397] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:54:57.815178 6292 tablet_bootstrap.cc:492] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: No bootstrap required, opened a new log
I20260927 10:54:57.815256 6291 ts_tablet_manager.cc:1397] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:54:57.815340 6292 ts_tablet_manager.cc:1397] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 10:54:57.818811 6291 raft_consensus.cc:348] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.819041 6291 raft_consensus.cc:374] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:57.819089 6291 raft_consensus.cc:729] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b9f4efa8ed44a6b0b42a07f8e1d83c, State: Initialized, Role: FOLLOWER
I20260927 10:54:57.819582 6291 consensus_queue.cc:260] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.819640 6290 raft_consensus.cc:348] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.819640 6292 raft_consensus.cc:348] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.819896 6290 raft_consensus.cc:374] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:57.819896 6292 raft_consensus.cc:374] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:57.819954 6290 raft_consensus.cc:729] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced6413f719e4e63b74d28de0924fec5, State: Initialized, Role: FOLLOWER
I20260927 10:54:57.819954 6292 raft_consensus.cc:729] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c30058d14894cf28d6e366c16dc19bc, State: Initialized, Role: FOLLOWER
I20260927 10:54:57.820330 6142 heartbeater.cc:503] Master 127.1.46.254:36305 was elected leader, sending a full tablet report...
I20260927 10:54:57.820393 6290 consensus_queue.cc:260] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.820399 6292 consensus_queue.cc:260] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.820583 6291 ts_tablet_manager.cc:1428] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:54:57.821360 6292 ts_tablet_manager.cc:1428] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 10:54:57.821413 6290 ts_tablet_manager.cc:1428] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:54:57.821676 6011 heartbeater.cc:503] Master 127.1.46.254:36305 was elected leader, sending a full tablet report...
I20260927 10:54:57.821686 6273 heartbeater.cc:503] Master 127.1.46.254:36305 was elected leader, sending a full tablet report...
W20260927 10:54:57.850905 6143 tablet.cc:2369] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:54:57.872332 6296 raft_consensus.cc:482] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:54:57.872501 6296 raft_consensus.cc:504] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.873557 6296 leader_election.cc:288] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c30058d14894cf28d6e366c16dc19bc (127.1.46.195:46369), ced6413f719e4e63b74d28de0924fec5 (127.1.46.193:42085)
I20260927 10:54:57.875401 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.873739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:46369 (local address 127.1.46.194:41933)
0927 10:54:57.873857 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.873874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.873952 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.874196 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.874201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.874218 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.874801 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.874808 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.874817 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.874864 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.875133 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.875135 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.875137 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.875223 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.875273 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 10:54:57.875411 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.873818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:41933 (local address 127.1.46.195:46369)
0927 10:54:57.873949 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.873956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.873974 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:54:57.874024 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.874027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.874068 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.874159 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.874979 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.874982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.874984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.875015 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.875040 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.875052 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.875144 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.875234 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.875285 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:54:57.875450 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.874028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:58005 (local address 127.1.46.193:42085)
0927 10:54:57.874165 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.874171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.874296 (+ 125us) server_negotiation.cc:407] Connection header received
0927 10:54:57.874394 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.874401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.874440 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.874521 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.875070 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.875076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.875082 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.875097 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.875119 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.875128 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.875212 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.875311 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.875342 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:54:57.875430 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.873810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:42085 (local address 127.1.46.194:58005)
0927 10:54:57.874192 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.874205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:57.874290 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:57.874548 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:57.874550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:57.874561 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:54:57.874917 (+ 356us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:54:57.874921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:54:57.874931 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:54:57.874956 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:54:57.875208 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:54:57.875211 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:54:57.875213 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 10:54:57.875283 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:54:57.875306 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":121,"threads_started":1}
I20260927 10:54:57.876080 6223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9353cbe32de74f20aff9037c00e2e8ee" candidate_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c30058d14894cf28d6e366c16dc19bc" is_pre_election: true
I20260927 10:54:57.876197 5965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9353cbe32de74f20aff9037c00e2e8ee" candidate_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced6413f719e4e63b74d28de0924fec5" is_pre_election: true
I20260927 10:54:57.876384 6223 raft_consensus.cc:2454] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b9f4efa8ed44a6b0b42a07f8e1d83c in term 0.
I20260927 10:54:57.876505 5965 raft_consensus.cc:2454] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b9f4efa8ed44a6b0b42a07f8e1d83c in term 0.
I20260927 10:54:57.876860 6031 leader_election.cc:302] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c30058d14894cf28d6e366c16dc19bc, c8b9f4efa8ed44a6b0b42a07f8e1d83c; no voters:
I20260927 10:54:57.877108 6296 raft_consensus.cc:2781] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:54:57.877183 6296 raft_consensus.cc:482] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:54:57.877429 6296 raft_consensus.cc:3037] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:57.878450 6296 raft_consensus.cc:504] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.878813 6296 leader_election.cc:288] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [CANDIDATE]: Term 1 election: Requested vote from peers 8c30058d14894cf28d6e366c16dc19bc (127.1.46.195:46369), ced6413f719e4e63b74d28de0924fec5 (127.1.46.193:42085)
I20260927 10:54:57.879180 6223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9353cbe32de74f20aff9037c00e2e8ee" candidate_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c30058d14894cf28d6e366c16dc19bc"
I20260927 10:54:57.879215 5965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9353cbe32de74f20aff9037c00e2e8ee" candidate_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced6413f719e4e63b74d28de0924fec5"
I20260927 10:54:57.879323 5965 raft_consensus.cc:3037] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:57.879323 6223 raft_consensus.cc:3037] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:57.880574 5965 raft_consensus.cc:2454] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b9f4efa8ed44a6b0b42a07f8e1d83c in term 1.
I20260927 10:54:57.880574 6223 raft_consensus.cc:2454] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b9f4efa8ed44a6b0b42a07f8e1d83c in term 1.
I20260927 10:54:57.880941 6032 leader_election.cc:302] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8c30058d14894cf28d6e366c16dc19bc, c8b9f4efa8ed44a6b0b42a07f8e1d83c, ced6413f719e4e63b74d28de0924fec5; no voters:
I20260927 10:54:57.881116 6296 raft_consensus.cc:2781] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:57.881351 6296 raft_consensus.cc:686] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 1 LEADER]: Becoming Leader. State: Replica: c8b9f4efa8ed44a6b0b42a07f8e1d83c, State: Running, Role: LEADER
I20260927 10:54:57.881649 6296 consensus_queue.cc:237] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } }
I20260927 10:54:57.884750 5821 catalog_manager.cc:5495] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c reported cstate change: term changed from 0 to 1, leader changed from <none> to c8b9f4efa8ed44a6b0b42a07f8e1d83c (127.1.46.194). New cstate: current_term: 1 leader_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b9f4efa8ed44a6b0b42a07f8e1d83c" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 35365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20260927 10:54:57.894712 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.893171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.1.46.194:35365)
0927 10:54:57.893286 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:57.893291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:57.893352 (+ 61us) server_negotiation.cc:407] Connection header received
0927 10:54:57.893531 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:57.893541 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:57.893592 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:57.893696 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:57.894290 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:57.894295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:57.894298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:57.894310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:57.894328 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:57.894338 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:57.894432 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:57.894511 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 10:54:57.894548 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:54:57.900197 6223 raft_consensus.cc:1261] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 1 FOLLOWER]: Refusing update from remote peer c8b9f4efa8ed44a6b0b42a07f8e1d83c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:57.900197 5965 raft_consensus.cc:1261] T 9353cbe32de74f20aff9037c00e2e8ee P ced6413f719e4e63b74d28de0924fec5 [term 1 FOLLOWER]: Refusing update from remote peer c8b9f4efa8ed44a6b0b42a07f8e1d83c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:54:57.900902 6300 consensus_queue.cc:1035] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced6413f719e4e63b74d28de0924fec5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:57.901077 6296 consensus_queue.cc:1035] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c30058d14894cf28d6e366c16dc19bc" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:54:57.907564 6305 mvcc.cc:205] Tried to move back new op lower bound from 7333914615392055296 to 7333914615325433856. Current Snapshot: MvccSnapshot[applied={T|T < 7333914615392055296}]
I20260927 10:54:57.908459 6303 mvcc.cc:205] Tried to move back new op lower bound from 7333914615392055296 to 7333914615325433856. Current Snapshot: MvccSnapshot[applied={T|T < 7333914615392055296}]
I20260927 10:54:57.908601 6304 mvcc.cc:205] Tried to move back new op lower bound from 7333914615392055296 to 7333914615325433856. Current Snapshot: MvccSnapshot[applied={T|T < 7333914615392055296}]
I20260927 10:54:57.916451 5823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:57.916576 5823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:54:57.918731 5823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 216c1585bcfa4760946a5f2ef575ed4f: Sending DeleteTablet for 3 replicas of tablet 9353cbe32de74f20aff9037c00e2e8ee
I20260927 10:54:57.919507 6207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9353cbe32de74f20aff9037c00e2e8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:57 UTC) from {username='slave'} at 127.0.0.1:44366
I20260927 10:54:57.919526 6076 tablet_service.cc:1514] Processing DeleteTablet for tablet 9353cbe32de74f20aff9037c00e2e8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:57 UTC) from {username='slave'} at 127.0.0.1:46796
I20260927 10:54:57.919999 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 5883
I20260927 10:54:57.920181 6312 tablet_replica.cc:330] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: stopping tablet replica
I20260927 10:54:57.920140 5945 tablet_service.cc:1514] Processing DeleteTablet for tablet 9353cbe32de74f20aff9037c00e2e8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:54:57 UTC) from {username='slave'} at 127.0.0.1:41102
I20260927 10:54:57.920477 6312 raft_consensus.cc:2229] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:54:57.920809 6312 raft_consensus.cc:2258] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:57.921128 6313 tablet_replica.cc:330] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: stopping tablet replica
I20260927 10:54:57.921396 6313 raft_consensus.cc:2229] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:54:57.921646 6313 raft_consensus.cc:2258] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:54:57.922269 6312 ts_tablet_manager.cc:1905] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:57.923023 6313 ts_tablet_manager.cc:1905] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:54:57.925587 6313 ts_tablet_manager.cc:1918] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:57.925621 6312 ts_tablet_manager.cc:1918] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:54:57.925680 6313 log.cc:1196] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/9353cbe32de74f20aff9037c00e2e8ee
I20260927 10:54:57.925707 6312 log.cc:1196] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/9353cbe32de74f20aff9037c00e2e8ee
I20260927 10:54:57.926030 6313 ts_tablet_manager.cc:1939] T 9353cbe32de74f20aff9037c00e2e8ee P 8c30058d14894cf28d6e366c16dc19bc: Deleting consensus metadata
I20260927 10:54:57.926070 6312 ts_tablet_manager.cc:1939] T 9353cbe32de74f20aff9037c00e2e8ee P c8b9f4efa8ed44a6b0b42a07f8e1d83c: Deleting consensus metadata
I20260927 10:54:57.927227 5809 catalog_manager.cc:4841] TS 8c30058d14894cf28d6e366c16dc19bc (127.1.46.195:46369): tablet 9353cbe32de74f20aff9037c00e2e8ee (table test-table [id=520ffedc6a2f41558e4a0936f95d8d62]) successfully deleted
I20260927 10:54:57.927237 5810 catalog_manager.cc:4841] TS c8b9f4efa8ed44a6b0b42a07f8e1d83c (127.1.46.194:35365): tablet 9353cbe32de74f20aff9037c00e2e8ee (table test-table [id=520ffedc6a2f41558e4a0936f95d8d62]) successfully deleted
W20260927 10:54:57.929924 5810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:42085 (error 108)
I20260927 10:54:57.930054 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6014
I20260927 10:54:57.932240 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:57.931971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:42085 (local address 127.0.0.1:41110)
0927 10:54:57.932097 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:57.932164 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 10:54:57.932474 5810 catalog_manager.cc:4584] TS ced6413f719e4e63b74d28de0924fec5 (127.1.46.193:42085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9353cbe32de74f20aff9037c00e2e8ee: Network error: Client connection negotiation failed: client connection to 127.1.46.193:42085: connect: Connection refused (error 111)
I20260927 10:54:57.935029 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6145
I20260927 10:54:57.940568 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 5792
2026-09-27T10:54:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (920 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 10:54:57.961833 1211 mini_oidc.cc:152] Starting JWKS server
I20260927 10:54:57.963472 1211 webserver.cc:460] Webserver started at https://127.0.0.1:37813/ using document root <none> and password file <none>
I20260927 10:54:57.963692 1211 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:54:57.963982 1211 webserver.cc:460] Webserver started at http://127.0.0.1:36437/ using document root <none> and password file <none>
2026-09-27T10:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:57Z Disabled control of system clock
I20260927 10:54:57.972725 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:46549
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:35025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:46549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37813/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/testchainca.cert with env {}
W20260927 10:54:58.080709 6320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:58.080982 6320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:58.081030 6320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:58.085327 6320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:58.085418 6320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:58.085435 6320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:58.085458 6320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:58.085476 6320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:54:58.085498 6320 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:54:58.085512 6320 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37813/jwks.json
I20260927 10:54:58.091356 6320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:35025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:46549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37813/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:58.092455 6320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:58.093487 6320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:58.101905 6325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:58.101859 6326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:58.101948 6320 server_base.cc:1034] running on GCE node
W20260927 10:54:58.101895 6328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:58.102695 6320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:58.103751 6320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:58.104915 6320 hybrid_clock.cc:648] HybridClock initialized: now 1790506498104896 us; error 30 us; skew 500 ppm
I20260927 10:54:58.107048 6320 webserver.cc:460] Webserver started at http://127.1.46.254:40107/ using document root <none> and password file <none>
I20260927 10:54:58.107678 6320 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:58.107785 6320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:58.108001 6320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:58.110225 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "abb717e652904b758b5bbab4915270ef"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "b999f13fbf2fd87b2cbe71b81972691c"
server_key_iv: "1a1447de1bcaf0d85e07b1ee86ee4e36"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.110912 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "abb717e652904b758b5bbab4915270ef"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "b999f13fbf2fd87b2cbe71b81972691c"
server_key_iv: "1a1447de1bcaf0d85e07b1ee86ee4e36"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.115137 6320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:54:58.117813 6334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:58.119225 6320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:54:58.119356 6320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "abb717e652904b758b5bbab4915270ef"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "b999f13fbf2fd87b2cbe71b81972691c"
server_key_iv: "1a1447de1bcaf0d85e07b1ee86ee4e36"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.119475 6320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:58.131897 6320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:58.132530 6320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:58.132717 6320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:58.139473 6320 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:46549
I20260927 10:54:58.139487 6386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:46549 every 8 connection(s)
I20260927 10:54:58.140377 6320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:58.143530 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:58.148211 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6320
I20260927 10:54:58.148347 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:58.148602 1211 external_mini_cluster.cc:1428] Setting key 93b3db159505f25106945b9233584336
I20260927 10:54:58.148862 6387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef: Bootstrap starting.
I20260927 10:54:58.151422 6387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:58.152125 6387 log.cc:824] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:58.153884 6387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef: No bootstrap required, opened a new log
I20260927 10:54:58.157371 6387 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb717e652904b758b5bbab4915270ef" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46549 } }
I20260927 10:54:58.157608 6387 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:58.157656 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:58.149931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.1.46.254:46549)
0927 10:54:58.150619 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:58.150643 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:58.150677 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:54:58.151305 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:58.151321 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:58.151742 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:58.152072 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:58.152484 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.153389 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.154364 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.154886 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.155052 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:58.155922 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:58.155927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:58.155950 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:58.156329 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:58.156384 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:58.156403 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:58.156506 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:58.156661 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 10:54:58.156906 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260927 10:54:58.157686 6387 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb717e652904b758b5bbab4915270ef, State: Initialized, Role: FOLLOWER
I20260927 10:54:58.158188 6387 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb717e652904b758b5bbab4915270ef" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46549 } }
I20260927 10:54:58.158339 6387 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:58.158398 6387 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:58.158483 6387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:58.160395 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:58.169003 6387 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb717e652904b758b5bbab4915270ef" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46549 } }
I20260927 10:54:58.169442 6387 leader_election.cc:302] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb717e652904b758b5bbab4915270ef; no voters:
I20260927 10:54:58.169782 6387 leader_election.cc:288] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:58.169901 6392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:58.170156 6392 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [term 1 LEADER]: Becoming Leader. State: Replica: abb717e652904b758b5bbab4915270ef, State: Running, Role: LEADER
I20260927 10:54:58.170509 6392 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb717e652904b758b5bbab4915270ef" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46549 } }
I20260927 10:54:58.171202 6387 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:58.172276 6393 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb717e652904b758b5bbab4915270ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb717e652904b758b5bbab4915270ef" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46549 } } }
I20260927 10:54:58.172413 6393 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:58.172699 6394 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb717e652904b758b5bbab4915270ef. Latest consensus state: current_term: 1 leader_uuid: "abb717e652904b758b5bbab4915270ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb717e652904b758b5bbab4915270ef" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46549 } } }
I20260927 10:54:58.172785 6394 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef [sys.catalog]: This master's current role is: LEADER
W20260927 10:54:58.176400 6407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:58.176491 6407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:58.176558 6401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:58.177261 6401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:58.182546 6401 catalog_manager.cc:1347] Generated new cluster ID: 0d34cbab020e4c0ebd5d05d9557a9333
I20260927 10:54:58.182634 6401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:58.207108 6401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:58.208074 6401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:58.216771 6401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abb717e652904b758b5bbab4915270ef: Generated new TSK 0
I20260927 10:54:58.217469 6401 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:54:58.254772 6417 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 10:54:58.254897 6417 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:54:58.255389 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:58.238415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.1.46.254:46549)
0927 10:54:58.238590 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:58.238595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:58.238857 (+ 262us) server_negotiation.cc:407] Connection header received
0927 10:54:58.239072 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:58.239074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:58.239123 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:58.239244 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:54:58.239515 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.240065 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.241347 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.241562 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.241655 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:58.242011 (+ 356us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:54:58.255057 (+ 13046us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:54:58.255276 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51404: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 10:54:58.255481 6390 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51404: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 10:54:58.256803 6416 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:58.238350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:46549 (local address 127.0.0.1:51404)
0927 10:54:58.238763 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:58.238779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:58.238909 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:58.239274 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:58.239277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:58.239285 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:54:58.239410 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:58.239417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.240193 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.240197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:58.241131 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:58.241140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.241675 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.241679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:58.241771 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:58.241838 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:54:58.255271 (+ 13433us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:54:58.255510 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.46.254:46549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":133,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.46.254:46549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 10:54:58.257519 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6320
2026-09-27T10:54:58Z chronyd exiting
I20260927 10:54:58.294904 1211 test_util.cc:175] -----------------------------------------------
I20260927 10:54:58.294987 1211 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790506472751632-1211-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 10:54:58.295970 1211 mini_oidc.cc:152] Starting JWKS server
I20260927 10:54:58.296818 1211 webserver.cc:460] Webserver started at https://127.0.0.1:36087/ using document root <none> and password file <none>
I20260927 10:54:58.296933 1211 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:54:58.297204 1211 webserver.cc:460] Webserver started at http://127.0.0.1:38019/ using document root <none> and password file <none>
2026-09-27T10:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:58Z Disabled control of system clock
I20260927 10:54:58.305760 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43211
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:42177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:43211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36087/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/testchainca.cert with env {}
W20260927 10:54:58.416374 6423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:58.416673 6423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:58.416754 6423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:58.420892 6423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:58.420970 6423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:58.421000 6423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:58.421054 6423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:58.421092 6423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:54:58.421129 6423 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:54:58.421161 6423 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36087/jwks.json
I20260927 10:54:58.426445 6423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:43211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36087/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:58.427474 6423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:58.428442 6423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:58.436582 6428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:58.436664 6431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:58.436628 6429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:58.437094 6423 server_base.cc:1034] running on GCE node
I20260927 10:54:58.437592 6423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:58.438656 6423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:58.439808 6423 hybrid_clock.cc:648] HybridClock initialized: now 1790506498439770 us; error 50 us; skew 500 ppm
I20260927 10:54:58.441572 6423 webserver.cc:460] Webserver started at http://127.1.46.254:34227/ using document root <none> and password file <none>
I20260927 10:54:58.442119 6423 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:58.442180 6423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:58.442391 6423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:58.444410 6423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "dcf8e62bfc574b539f25a63c37723030"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "d8ba834791cf3262f0bbf3660d4094d2"
server_key_iv: "26cff472d5c94a884064cb278328af15"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.444952 6423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "dcf8e62bfc574b539f25a63c37723030"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "d8ba834791cf3262f0bbf3660d4094d2"
server_key_iv: "26cff472d5c94a884064cb278328af15"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.448498 6423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:54:58.450798 6437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:58.451941 6423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:54:58.452090 6423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "dcf8e62bfc574b539f25a63c37723030"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "d8ba834791cf3262f0bbf3660d4094d2"
server_key_iv: "26cff472d5c94a884064cb278328af15"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.452196 6423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:58.477797 6423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:58.478665 6423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:58.478879 6423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:58.485615 6423 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:43211
I20260927 10:54:58.485615 6489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:43211 every 8 connection(s)
I20260927 10:54:58.486538 6423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:58.489567 6490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:58.491389 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6423
I20260927 10:54:58.491500 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:58.491787 1211 external_mini_cluster.cc:1428] Setting key f290a96dbbe51848da91d94c276abef8
I20260927 10:54:58.495610 6490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030: Bootstrap starting.
I20260927 10:54:58.497747 6490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:58.498445 6490 log.cc:824] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:58.500150 6490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030: No bootstrap required, opened a new log
I20260927 10:54:58.501080 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:58.493099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.1.46.254:43211)
0927 10:54:58.494038 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:58.494053 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:58.494086 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:54:58.494823 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:58.494842 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:58.495226 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:58.495566 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:58.495975 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.496949 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.497968 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.498438 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.498644 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:58.499475 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:58.499480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:58.499491 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:58.499849 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:58.499898 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:58.499917 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:58.500018 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:58.500140 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 10:54:58.500430 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":121,"threads_started":1}
I20260927 10:54:58.503947 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:58.504070 6490 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf8e62bfc574b539f25a63c37723030" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43211 } }
I20260927 10:54:58.504310 6490 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:58.504360 6490 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf8e62bfc574b539f25a63c37723030, State: Initialized, Role: FOLLOWER
I20260927 10:54:58.504807 6490 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf8e62bfc574b539f25a63c37723030" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43211 } }
I20260927 10:54:58.504964 6490 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:58.505048 6490 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:58.505151 6490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:58.576479 6490 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf8e62bfc574b539f25a63c37723030" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43211 } }
I20260927 10:54:58.576997 6490 leader_election.cc:302] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcf8e62bfc574b539f25a63c37723030; no voters:
I20260927 10:54:58.577412 6490 leader_election.cc:288] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:58.577557 6495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:58.577796 6495 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [term 1 LEADER]: Becoming Leader. State: Replica: dcf8e62bfc574b539f25a63c37723030, State: Running, Role: LEADER
I20260927 10:54:58.578163 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf8e62bfc574b539f25a63c37723030" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43211 } }
I20260927 10:54:58.578889 6490 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:58.579969 6496 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcf8e62bfc574b539f25a63c37723030" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf8e62bfc574b539f25a63c37723030" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43211 } } }
I20260927 10:54:58.580094 6496 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:58.580713 6497 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf8e62bfc574b539f25a63c37723030. Latest consensus state: current_term: 1 leader_uuid: "dcf8e62bfc574b539f25a63c37723030" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf8e62bfc574b539f25a63c37723030" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43211 } } }
I20260927 10:54:58.580833 6497 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030 [sys.catalog]: This master's current role is: LEADER
W20260927 10:54:58.584331 6510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:58.584412 6510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:58.584506 6504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:58.585330 6504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:58.590857 6504 catalog_manager.cc:1347] Generated new cluster ID: 46130b0b0b7444228ee4c79cc69afe42
I20260927 10:54:58.590930 6504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:58.619107 6504 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:58.620111 6504 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:58.625425 6504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcf8e62bfc574b539f25a63c37723030: Generated new TSK 0
I20260927 10:54:58.626127 6504 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:54:58.649212 6520 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 10:54:58.649295 6520 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:54:58.649994 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:58.633345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.1.46.254:43211)
0927 10:54:58.633480 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:58.633485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:58.633680 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:54:58.633884 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:58.633886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:58.633931 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:58.634035 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:54:58.634284 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.634881 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.635913 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.636134 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.636243 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:58.636548 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:54:58.649570 (+ 13022us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:54:58.649830 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 10:54:58.650049 6519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:58.633276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:43211 (local address 127.0.0.1:38812)
0927 10:54:58.633595 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:58.633607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:58.633734 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:58.634046 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:58.634049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:58.634057 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:54:58.634173 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:58.634178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.635011 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.635016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:58.635800 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:58.635809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.636246 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.636250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:58.636351 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:58.636382 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:54:58.649739 (+ 13357us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:54:58.649885 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.46.254:43211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
W20260927 10:54:58.650061 6493 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 10:54:58.650882 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6423
2026-09-27T10:54:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (393 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 10:54:58.689764 1211 mini_oidc.cc:152] Starting JWKS server
I20260927 10:54:58.690689 1211 webserver.cc:460] Webserver started at https://127.0.0.1:40281/ using document root <none> and password file <none>
I20260927 10:54:58.690775 1211 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:54:58.691043 1211 webserver.cc:460] Webserver started at http://127.0.0.1:34617/ using document root <none> and password file <none>
2026-09-27T10:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:58Z Disabled control of system clock
I20260927 10:54:58.699931 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:46287
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:46287
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40281/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:54:58.829258 6526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:58.829525 6526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:58.829571 6526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:58.833551 6526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:58.833603 6526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:58.833618 6526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:58.833633 6526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:58.833648 6526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:54:58.833663 6526 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:54:58.833676 6526 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40281/jwks.json
I20260927 10:54:58.838680 6526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:46287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40281/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:58.839670 6526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:58.840986 6526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:58.849962 6532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:58.849972 6531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:58.849972 6534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:58.850055 6526 server_base.cc:1034] running on GCE node
I20260927 10:54:58.850793 6526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:58.851825 6526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:58.853016 6526 hybrid_clock.cc:648] HybridClock initialized: now 1790506498853004 us; error 26 us; skew 500 ppm
I20260927 10:54:58.854893 6526 webserver.cc:460] Webserver started at http://127.1.46.254:39185/ using document root <none> and password file <none>
I20260927 10:54:58.855468 6526 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:58.855556 6526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:58.855798 6526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:58.857772 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "6fb9f2825f1a483cb158ab49351178f8"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "e8b5e4c48b719a55e134569950fd6c6b"
server_key_iv: "fcacc4ef6ae21c60d97843fda8e73ea9"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.858354 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "6fb9f2825f1a483cb158ab49351178f8"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "e8b5e4c48b719a55e134569950fd6c6b"
server_key_iv: "fcacc4ef6ae21c60d97843fda8e73ea9"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.861917 6526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:54:58.864351 6540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:58.865602 6526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:54:58.865753 6526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "6fb9f2825f1a483cb158ab49351178f8"
format_stamp: "Formatted at 2026-09-27 10:54:58 on dist-test-slave-cffh"
server_key: "e8b5e4c48b719a55e134569950fd6c6b"
server_key_iv: "fcacc4ef6ae21c60d97843fda8e73ea9"
server_key_version: "encryptionkey@0"
I20260927 10:54:58.865866 6526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:58.888471 6526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:58.889223 6526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:58.889454 6526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:58.897082 6592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:46287 every 8 connection(s)
I20260927 10:54:58.897080 6526 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:46287
I20260927 10:54:58.898074 6526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:58.901292 6593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:58.905602 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6526
I20260927 10:54:58.905714 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:58.905975 1211 external_mini_cluster.cc:1428] Setting key c29fceeea15bb07fcb1e7cb37ad74641
I20260927 10:54:58.907215 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8: Bootstrap starting.
I20260927 10:54:58.909479 6593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:58.910336 6593 log.cc:824] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:58.912141 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8: No bootstrap required, opened a new log
I20260927 10:54:58.915143 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:58.907370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.1.46.254:46287)
0927 10:54:58.907975 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:58.907993 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:58.908028 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:54:58.908625 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:58.908641 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:58.909018 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:58.909291 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:58.909723 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.910831 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.911726 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:58.912215 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:58.912399 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:58.913340 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:58.913345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:58.913356 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:58.913742 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:58.913797 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:58.913817 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:58.913926 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:58.914111 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 10:54:58.914366 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260927 10:54:58.915725 6593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb9f2825f1a483cb158ab49351178f8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46287 } }
I20260927 10:54:58.915980 6593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:58.916045 6593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb9f2825f1a483cb158ab49351178f8, State: Initialized, Role: FOLLOWER
I20260927 10:54:58.916481 6593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb9f2825f1a483cb158ab49351178f8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46287 } }
I20260927 10:54:58.916678 6593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:58.916771 6593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:58.916877 6593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:58.918048 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:58.984287 6593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb9f2825f1a483cb158ab49351178f8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46287 } }
I20260927 10:54:58.984794 6593 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb9f2825f1a483cb158ab49351178f8; no voters:
I20260927 10:54:58.985174 6593 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:58.985322 6598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:58.985555 6598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [term 1 LEADER]: Becoming Leader. State: Replica: 6fb9f2825f1a483cb158ab49351178f8, State: Running, Role: LEADER
I20260927 10:54:58.985913 6598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb9f2825f1a483cb158ab49351178f8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46287 } }
I20260927 10:54:58.986639 6593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:58.987617 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb9f2825f1a483cb158ab49351178f8. Latest consensus state: current_term: 1 leader_uuid: "6fb9f2825f1a483cb158ab49351178f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb9f2825f1a483cb158ab49351178f8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46287 } } }
I20260927 10:54:58.987766 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:58.988600 6599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb9f2825f1a483cb158ab49351178f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb9f2825f1a483cb158ab49351178f8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46287 } } }
I20260927 10:54:58.988715 6599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8 [sys.catalog]: This master's current role is: LEADER
W20260927 10:54:58.991801 6613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:54:58.991886 6613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:54:58.991961 6611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:58.992710 6611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:58.998003 6611 catalog_manager.cc:1347] Generated new cluster ID: 80c5ac77d717401da65df6551dd5066e
I20260927 10:54:58.998067 6611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:59.012116 6611 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:59.013111 6611 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:59.020941 6611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb9f2825f1a483cb158ab49351178f8: Generated new TSK 0
I20260927 10:54:59.021625 6611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:59.043049 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:59.038090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.1.46.254:46287)
0927 10:54:59.038242 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:59.038246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:59.038454 (+ 208us) server_negotiation.cc:407] Connection header received
0927 10:54:59.038660 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:59.038663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:59.038707 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:59.038804 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:54:59.039073 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.039628 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.040852 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.041062 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.041132 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:59.041442 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:54:59.042667 (+ 1225us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40281/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:54:59.042901 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40281/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 10:54:59.043123 6622 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:59.038037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:46287 (local address 127.0.0.1:34968)
0927 10:54:59.038352 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:59.038379 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:59.038490 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:59.038828 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:59.038831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:59.038840 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:54:59.038971 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:59.038977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.039740 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.039744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:59.040737 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:59.040747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.041169 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.041173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:59.041261 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:59.041286 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:54:59.042828 (+ 1542us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40281/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:54:59.042959 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.46.254:46287: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40281/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":127,"threads_started":1}
W20260927 10:54:59.043116 6596 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40281/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 10:54:59.043869 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6526
E20260927 10:54:59.055867 6623 webserver.cc:533] Webserver: sslize failed:
E20260927 10:54:59.055929 6623 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T10:54:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (393 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T10:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:54:59Z Disabled control of system clock
I20260927 10:54:59.093052 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:46567
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:46567
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/testchainca.cert with env {}
W20260927 10:54:59.207265 6627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:54:59.207533 6627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:54:59.207573 6627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:54:59.211676 6627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:54:59.211737 6627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:54:59.211752 6627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:54:59.211769 6627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:54:59.211791 6627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:54:59.211846 6627 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:54:59.211876 6627 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 10:54:59.217258 6627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:46567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:54:59.218322 6627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:54:59.219374 6627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:54:59.228397 6633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:59.228541 6627 server_base.cc:1034] running on GCE node
W20260927 10:54:59.228405 6635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:54:59.228405 6632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:54:59.229221 6627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:54:59.230327 6627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:54:59.231506 6627 hybrid_clock.cc:648] HybridClock initialized: now 1790506499231490 us; error 34 us; skew 500 ppm
I20260927 10:54:59.233294 6627 webserver.cc:460] Webserver started at http://127.1.46.254:33601/ using document root <none> and password file <none>
I20260927 10:54:59.233880 6627 fs_manager.cc:359] Metadata directory not provided
I20260927 10:54:59.233970 6627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:54:59.234186 6627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:54:59.236266 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "7f4815af37bf423f9a90f7a409525036"
format_stamp: "Formatted at 2026-09-27 10:54:59 on dist-test-slave-cffh"
server_key: "85d032dba4540e30dd553e349e4146c4"
server_key_iv: "0e8ca4dfa13bde0dd02f24bd883e1ef7"
server_key_version: "encryptionkey@0"
I20260927 10:54:59.236850 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "7f4815af37bf423f9a90f7a409525036"
format_stamp: "Formatted at 2026-09-27 10:54:59 on dist-test-slave-cffh"
server_key: "85d032dba4540e30dd553e349e4146c4"
server_key_iv: "0e8ca4dfa13bde0dd02f24bd883e1ef7"
server_key_version: "encryptionkey@0"
I20260927 10:54:59.240399 6627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 10:54:59.242717 6641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:54:59.243916 6627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:54:59.244088 6627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "7f4815af37bf423f9a90f7a409525036"
format_stamp: "Formatted at 2026-09-27 10:54:59 on dist-test-slave-cffh"
server_key: "85d032dba4540e30dd553e349e4146c4"
server_key_iv: "0e8ca4dfa13bde0dd02f24bd883e1ef7"
server_key_version: "encryptionkey@0"
I20260927 10:54:59.244202 6627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:54:59.263139 6627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:54:59.263820 6627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:54:59.264024 6627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:54:59.270766 6627 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:46567
I20260927 10:54:59.270761 6693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:46567 every 8 connection(s)
I20260927 10:54:59.271786 6627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:54:59.274893 6694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:54:59.278786 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6627
I20260927 10:54:59.278882 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:54:59.279156 1211 external_mini_cluster.cc:1428] Setting key affa18f18e7e241af77f141eb46b6cee
I20260927 10:54:59.280716 6694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036: Bootstrap starting.
I20260927 10:54:59.283367 6694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036: Neither blocks nor log segments found. Creating new log.
I20260927 10:54:59.284246 6694 log.cc:824] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036: Log is configured to *not* fsync() on all Append() calls
I20260927 10:54:59.286192 6694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036: No bootstrap required, opened a new log
I20260927 10:54:59.289691 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:59.280557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.1.46.254:46567)
0927 10:54:59.281208 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:59.281229 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:59.281269 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:54:59.282128 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:59.282151 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:59.282674 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:59.283017 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:54:59.283505 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.284950 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.285857 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.286541 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.286741 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:59.287574 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:54:59.287580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:54:59.287598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:54:59.288084 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:54:59.288150 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:54:59.288177 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:54:59.288279 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:54:59.288446 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 10:54:59.288710 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260927 10:54:59.290740 6694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4815af37bf423f9a90f7a409525036" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46567 } }
I20260927 10:54:59.291010 6694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:54:59.291074 6694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4815af37bf423f9a90f7a409525036, State: Initialized, Role: FOLLOWER
I20260927 10:54:59.291513 6694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4815af37bf423f9a90f7a409525036" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46567 } }
I20260927 10:54:59.291680 6694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:54:59.291751 6694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:54:59.291858 6694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:54:59.292867 6694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4815af37bf423f9a90f7a409525036" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46567 } }
I20260927 10:54:59.293191 6694 leader_election.cc:302] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f4815af37bf423f9a90f7a409525036; no voters:
I20260927 10:54:59.293525 6694 leader_election.cc:288] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:54:59.293648 6699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:54:59.293907 6699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [term 1 LEADER]: Becoming Leader. State: Replica: 7f4815af37bf423f9a90f7a409525036, State: Running, Role: LEADER
I20260927 10:54:59.294229 6699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4815af37bf423f9a90f7a409525036" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46567 } }
I20260927 10:54:59.294788 6694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:54:59.295740 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:54:59.296118 6700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f4815af37bf423f9a90f7a409525036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4815af37bf423f9a90f7a409525036" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46567 } } }
I20260927 10:54:59.296267 6700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:59.296663 6710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:54:59.296622 6701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f4815af37bf423f9a90f7a409525036. Latest consensus state: current_term: 1 leader_uuid: "7f4815af37bf423f9a90f7a409525036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4815af37bf423f9a90f7a409525036" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 46567 } } }
I20260927 10:54:59.297135 6701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036 [sys.catalog]: This master's current role is: LEADER
I20260927 10:54:59.299881 6710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:54:59.305083 6710 catalog_manager.cc:1347] Generated new cluster ID: 70b88ad888b6469cb9bf4b7b365f7006
I20260927 10:54:59.305146 6710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:54:59.323546 6710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:54:59.324542 6710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:54:59.329857 6710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f4815af37bf423f9a90f7a409525036: Generated new TSK 0
I20260927 10:54:59.330545 6710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:54:59.397306 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:54:59.345620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.1.46.254:46567)
0927 10:54:59.345765 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:54:59.345770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:54:59.345916 (+ 146us) server_negotiation.cc:407] Connection header received
0927 10:54:59.346043 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:54:59.346047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:54:59.346107 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:54:59.346233 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:54:59.346485 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.347355 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.348403 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.348784 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.348891 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:59.349184 (+ 293us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:54:59.396780 (+ 47596us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:54:59.397136 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46586: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:54:59.397408 6697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46586: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 10:54:59.397413 6723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:54:59.345488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:46567 (local address 127.0.0.1:46586)
0927 10:54:59.345830 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 10:54:59.345842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:54:59.345934 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:54:59.346251 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:54:59.346254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:54:59.346261 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:54:59.346369 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:59.346374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.347521 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.347524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:59.348265 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:54:59.348273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:54:59.348928 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:54:59.348931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:54:59.348994 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:54:59.349015 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:54:59.397031 (+ 48016us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:54:59.397221 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.46.254:46567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":109,"threads_started":1}
I20260927 10:54:59.398304 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6627
2026-09-27T10:54:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (333 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (132 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:54:59 dist-test-slave-cffh krb5kdc[6735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:54:59 dist-test-slave-cffh krb5kdc[6735](info): set up 2 sockets
Sep 27 10:54:59 dist-test-slave-cffh krb5kdc[6735](info): commencing operation
krb5kdc: starting...
W20260927 10:55:00.967010 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:55:01.079571 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:37611
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:37611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:01.187482 6748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:01.187781 6748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:01.187857 6748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:01.191766 6748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:01.191852 6748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:01.191892 6748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:01.191949 6748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:01.191988 6748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:01.197451 6748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:37611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:37611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:01.198450 6748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:01.199450 6748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:01.207522 6756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:01.207537 6754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:01.207532 6753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:01.207592 6748 server_base.cc:1034] running on GCE node
I20260927 10:55:01.208334 6748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:01.209378 6748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:01.210635 6748 hybrid_clock.cc:648] HybridClock initialized: now 1790506501210600 us; error 45 us; skew 500 ppm
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:01.220093 6748 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:55:01.220965 6748 webserver.cc:460] Webserver started at http://127.1.46.254:45831/ using document root <none> and password file <none>
I20260927 10:55:01.221546 6748 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:01.221642 6748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:01.221881 6748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:01.223956 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "b9f55054d63647f6a01c8df2c63e9d43"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "7f71d3cb9bb2459bbac5edeaf1afb7ab"
server_key_iv: "926b7847eedde38eeb440a8902cc2d06"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.224562 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "b9f55054d63647f6a01c8df2c63e9d43"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "7f71d3cb9bb2459bbac5edeaf1afb7ab"
server_key_iv: "926b7847eedde38eeb440a8902cc2d06"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.228137 6748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:01.230445 6763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.231674 6748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:55:01.231829 6748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "b9f55054d63647f6a01c8df2c63e9d43"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "7f71d3cb9bb2459bbac5edeaf1afb7ab"
server_key_iv: "926b7847eedde38eeb440a8902cc2d06"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.231936 6748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:01.259461 6748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:01.262084 6748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:01.262305 6748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:01.269188 6748 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:37611
I20260927 10:55:01.269232 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:37611 every 8 connection(s)
I20260927 10:55:01.270136 6748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:01.272819 6816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:01.274921 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6748
I20260927 10:55:01.275049 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:55:01.275311 1211 external_mini_cluster.cc:1428] Setting key 555bf9e1b1986fb190efc7c0db859d81
I20260927 10:55:01.279162 6816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43: Bootstrap starting.
I20260927 10:55:01.281709 6816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:01.282547 6816 log.cc:824] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:01.284642 6816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43: No bootstrap required, opened a new log
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:01.288018 6816 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f55054d63647f6a01c8df2c63e9d43" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37611 } }
I20260927 10:55:01.288287 6816 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:01.288379 6816 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f55054d63647f6a01c8df2c63e9d43, State: Initialized, Role: FOLLOWER
I20260927 10:55:01.288887 6816 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f55054d63647f6a01c8df2c63e9d43" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37611 } }
I20260927 10:55:01.289009 6816 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:01.289089 6816 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:01.289203 6816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:01.290324 6816 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f55054d63647f6a01c8df2c63e9d43" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37611 } }
I20260927 10:55:01.290714 6816 leader_election.cc:302] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f55054d63647f6a01c8df2c63e9d43; no voters:
I20260927 10:55:01.290972 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:01.276695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.1.46.254:37611)
0927 10:55:01.277090 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:01.277110 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:01.277153 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:55:01.278100 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:01.278123 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:01.278538 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:01.278923 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:01.279647 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.281112 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.282008 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.282704 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.282871 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:01.285515 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:01.285519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:01.285532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:01.285557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:01.287784 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.288367 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.288377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.288386 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.288529 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.288824 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.288826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.288827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.289315 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:01.289477 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:01.289759 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 10:55:01.290053 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260927 10:55:01.291015 6816 leader_election.cc:288] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:01.291118 6821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:01.291332 6821 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [term 1 LEADER]: Becoming Leader. State: Replica: b9f55054d63647f6a01c8df2c63e9d43, State: Running, Role: LEADER
I20260927 10:55:01.291684 6821 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f55054d63647f6a01c8df2c63e9d43" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37611 } }
I20260927 10:55:01.292361 6816 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:01.292833 6821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f55054d63647f6a01c8df2c63e9d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f55054d63647f6a01c8df2c63e9d43" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37611 } } }
I20260927 10:55:01.292999 6821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:01.293397 6827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:01.294196 6822 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f55054d63647f6a01c8df2c63e9d43. Latest consensus state: current_term: 1 leader_uuid: "b9f55054d63647f6a01c8df2c63e9d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f55054d63647f6a01c8df2c63e9d43" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37611 } } }
I20260927 10:55:01.294296 6822 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:01.296525 6827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:01.301832 6827 catalog_manager.cc:1347] Generated new cluster ID: ce43fd1c15ba43a2a45c149d7413d18b
I20260927 10:55:01.301903 6827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:01.321146 6827 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:01.322333 6827 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:01.330794 6827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9f55054d63647f6a01c8df2c63e9d43: Generated new TSK 0
I20260927 10:55:01.331493 6827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:55:01.363758 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37611
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:01.475648 6842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:01.475910 6842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:01.475982 6842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:01.479969 6842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:01.480046 6842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:01.480134 6842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:01.485060 6842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:01.486042 6842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:01.487042 6842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:01.496129 6848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:01.496143 6847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:01.496143 6850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:01.496482 6842 server_base.cc:1034] running on GCE node
I20260927 10:55:01.496903 6842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:01.497488 6842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:01.498641 6842 hybrid_clock.cc:648] HybridClock initialized: now 1790506501498626 us; error 31 us; skew 500 ppm
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:01.501317 6842 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:55:01.502168 6842 webserver.cc:460] Webserver started at http://127.1.46.193:46513/ using document root <none> and password file <none>
I20260927 10:55:01.502755 6842 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:01.502851 6842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:01.503063 6842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:01.504940 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "79a225049db745c082e7954c9df93895"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "f9085deb85264600f1fac2da0c78b1a4"
server_key_iv: "2fc68acf831e7277403f66d8dd7fb99a"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.505487 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "79a225049db745c082e7954c9df93895"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "f9085deb85264600f1fac2da0c78b1a4"
server_key_iv: "2fc68acf831e7277403f66d8dd7fb99a"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.508955 6842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:55:01.511248 6857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.512461 6842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:55:01.512588 6842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "79a225049db745c082e7954c9df93895"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "f9085deb85264600f1fac2da0c78b1a4"
server_key_iv: "2fc68acf831e7277403f66d8dd7fb99a"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.512689 6842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:01.526839 6842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:01.530014 6842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:01.530216 6842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:01.530804 6842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:01.531579 6842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:01.531648 6842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.531728 6842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:01.531778 6842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.540302 6842 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:45303
I20260927 10:55:01.540328 6970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:45303 every 8 connection(s)
I20260927 10:55:01.541153 6842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:01.550239 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6842
I20260927 10:55:01.550369 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:01.550700 1211 external_mini_cluster.cc:1428] Setting key d32277c1af0c6c2adbd0e8f026529b8e
I20260927 10:55:01.554244 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:01.542907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:41631 (local address 127.1.46.254:37611)
0927 10:55:01.543053 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:01.543057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:01.543929 (+ 872us) server_negotiation.cc:407] Connection header received
0927 10:55:01.545245 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:01.545248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:01.545300 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:01.545391 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:01.546473 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.547082 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.548168 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.548383 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.548458 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:01.550796 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:01.550800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:01.550804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:01.550837 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:01.552291 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.552789 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.552794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.552797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.552875 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.553198 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.553203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.553206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.553377 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:01.553484 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:01.553975 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 10:55:01.554084 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:55:01.555248 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:01.543347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37611 (local address 127.1.46.193:41631)
0927 10:55:01.543767 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:01.543805 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:01.545048 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:01.545542 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:01.545548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:01.545876 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:01.546318 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:01.546330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.547196 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.547198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:01.548070 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:01.548078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.548501 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.548503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:01.548635 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:01.549112 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:01.549127 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:01.550636 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:01.552428 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:01.552434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:01.552447 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:01.552686 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:01.552964 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:01.552969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:01.552971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:01.553105 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:01.553491 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:01.553499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:01.553743 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 10:55:01.553924 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 10:55:01.554204 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260927 10:55:01.556514 6971 heartbeater.cc:347] Connected to a master server at 127.1.46.254:37611
I20260927 10:55:01.556732 6971 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:01.557204 6971 heartbeater.cc:511] Master 127.1.46.254:37611 requested a full tablet report, sending...
I20260927 10:55:01.558826 6780 ts_manager.cc:194] Registered new tserver with Master: 79a225049db745c082e7954c9df93895 (127.1.46.193:45303)
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:55:01.560448 6780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:41631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:55:01.577659 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37611
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:01.683420 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:01.683660 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:01.683699 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:01.687569 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:01.687614 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:01.687669 6976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:01.692804 6976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:01.693806 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:01.694813 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:01.703802 6984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:01.703872 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:01.703794 6981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:01.704026 6976 server_base.cc:1034] running on GCE node
I20260927 10:55:01.704469 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:01.705000 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:01.706146 6976 hybrid_clock.cc:648] HybridClock initialized: now 1790506501706119 us; error 43 us; skew 500 ppm
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:01.708799 6976 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:55:01.709576 6976 webserver.cc:460] Webserver started at http://127.1.46.194:35945/ using document root <none> and password file <none>
I20260927 10:55:01.710047 6976 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:01.710101 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:01.710248 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:01.712141 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "2ab1650828904cc3851e0c4e777b238e"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "be14718b459f28648ea0d71fae11ecb7"
server_key_iv: "e6cb22a8c64d768c54ab00fe2de60508"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.712687 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "2ab1650828904cc3851e0c4e777b238e"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "be14718b459f28648ea0d71fae11ecb7"
server_key_iv: "e6cb22a8c64d768c54ab00fe2de60508"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.716034 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:55:01.718448 6991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.719695 6976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:01.719785 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "2ab1650828904cc3851e0c4e777b238e"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "be14718b459f28648ea0d71fae11ecb7"
server_key_iv: "e6cb22a8c64d768c54ab00fe2de60508"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.719861 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:01.733224 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:01.736366 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:01.736521 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:01.737011 6976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:01.737735 6976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:01.737778 6976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.737816 6976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:01.737831 6976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.746487 6976 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:34225
I20260927 10:55:01.746507 7104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:34225 every 8 connection(s)
I20260927 10:55:01.747400 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:01.753293 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 6976
I20260927 10:55:01.753387 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:01.753654 1211 external_mini_cluster.cc:1428] Setting key 943e5ba16fb5024ea48afd35843bc69d
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:01.760926 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:01.749090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:43519 (local address 127.1.46.254:37611)
0927 10:55:01.749247 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:01.749252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:01.750050 (+ 798us) server_negotiation.cc:407] Connection header received
0927 10:55:01.751330 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:01.751332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:01.751378 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:01.751461 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:01.752528 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.753081 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.754405 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.754705 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.754817 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:01.757368 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:01.757375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:01.757381 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:01.757409 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:01.758885 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.759469 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.759474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.759477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.759557 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.759894 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.759898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.759900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.760080 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:01.760186 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:01.760640 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 10:55:01.760754 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260927 10:55:01.761670 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:01.749480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37611 (local address 127.1.46.194:43519)
0927 10:55:01.749905 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:01.749945 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:01.751149 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:01.751607 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:01.751613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:01.751955 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:01.752404 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:01.752418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.753195 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.753197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:01.754287 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:01.754295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.754855 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.754860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:01.755061 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:01.755617 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:01.755640 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:01.757196 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:01.759033 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:01.759039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:01.759060 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:01.759366 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:01.759654 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:01.759657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:01.759660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:01.759792 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:01.760184 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:01.760191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:01.760420 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 10:55:01.760597 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:55:01.760889 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260927 10:55:01.762931 7105 heartbeater.cc:347] Connected to a master server at 127.1.46.254:37611
I20260927 10:55:01.763165 7105 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:55:01.763720 7105 heartbeater.cc:511] Master 127.1.46.254:37611 requested a full tablet report, sending...
I20260927 10:55:01.764780 6780 ts_manager.cc:194] Registered new tserver with Master: 2ab1650828904cc3851e0c4e777b238e (127.1.46.194:34225)
I20260927 10:55:01.765316 6780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:43519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:55:01.779516 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37611
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:01.886330 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:01.886628 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:01.886718 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:01.890656 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:01.890707 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:01.890813 7110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:01.896081 7110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:37611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:01.897034 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:01.898020 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:01.907347 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:01.907480 7110 server_base.cc:1034] running on GCE node
W20260927 10:55:01.907361 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:01.907380 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:01.908056 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:01.908654 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:01.909832 7110 hybrid_clock.cc:648] HybridClock initialized: now 1790506501909821 us; error 31 us; skew 500 ppm
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:01.912496 7110 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:55:01.913378 7110 webserver.cc:460] Webserver started at http://127.1.46.195:45497/ using document root <none> and password file <none>
I20260927 10:55:01.913873 7110 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:01.913931 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:01.914083 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:01.916164 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "c4c1bfc73195488ba91539f75572ba0c"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "d66a3395ba6c6c9c337be32f7ac41371"
server_key_iv: "3d74b046bb7b1d1671ef3abecfc6a10a"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.916649 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "c4c1bfc73195488ba91539f75572ba0c"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "d66a3395ba6c6c9c337be32f7ac41371"
server_key_iv: "3d74b046bb7b1d1671ef3abecfc6a10a"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.920212 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:01.922449 7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.923712 7110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:55:01.923796 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "c4c1bfc73195488ba91539f75572ba0c"
format_stamp: "Formatted at 2026-09-27 10:55:01 on dist-test-slave-cffh"
server_key: "d66a3395ba6c6c9c337be32f7ac41371"
server_key_iv: "3d74b046bb7b1d1671ef3abecfc6a10a"
server_key_version: "encryptionkey@0"
I20260927 10:55:01.923878 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:01.949309 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:01.952494 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:01.952639 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:01.953154 7110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:01.953879 7110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:01.953926 7110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.953963 7110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:01.953989 7110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:01.962515 7110 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:36333
I20260927 10:55:01.962541 7238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:36333 every 8 connection(s)
I20260927 10:55:01.963407 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:01.965097 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 7110
I20260927 10:55:01.965195 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:01.965483 1211 external_mini_cluster.cc:1428] Setting key fc4019bf904646b61951c90550ee395b
Sep 27 10:55:01 dist-test-slave-cffh krb5kdc[6735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506501, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:01.976867 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:01.965057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:51817 (local address 127.1.46.254:37611)
0927 10:55:01.965199 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:01.965204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:01.966142 (+ 938us) server_negotiation.cc:407] Connection header received
0927 10:55:01.967950 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:01.967953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:01.967995 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:01.968058 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:01.969246 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.969764 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.971001 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.971206 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.971293 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:01.973647 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:01.973650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:01.973652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:01.973675 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:01.975192 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.975642 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.975644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.975646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.975714 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:01.975991 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:01.975993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:01.975994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:01.976147 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:01.976220 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:01.976656 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 10:55:01.976753 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:55:01.977733 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:01.965545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:37611 (local address 127.1.46.195:51817)
0927 10:55:01.966015 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:01.966064 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:01.967758 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:01.968232 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:01.968239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:01.968642 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:01.969124 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:01.969137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.969880 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.969883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:01.970852 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:01.970861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:01.971375 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:01.971378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:01.971518 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:01.972038 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:01.972054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:01.973500 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:01.975298 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:01.975302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:01.975312 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:01.975557 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:01.975790 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:01.975792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:01.975793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:01.975914 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:01.976229 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:01.976233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:01.976458 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 10:55:01.976610 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 10:55:01.976827 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260927 10:55:01.979018 7239 heartbeater.cc:347] Connected to a master server at 127.1.46.254:37611
I20260927 10:55:01.979264 7239 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:01.979717 7239 heartbeater.cc:511] Master 127.1.46.254:37611 requested a full tablet report, sending...
I20260927 10:55:01.980639 6780 ts_manager.cc:194] Registered new tserver with Master: c4c1bfc73195488ba91539f75572ba0c (127.1.46.195:36333)
I20260927 10:55:01.981194 6780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:51817
I20260927 10:55:01.990959 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/test-admin.keytab.
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:02.006384 1211 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 10:55:02.009245 7246 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:02.007849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.1.46.193:45303)
0927 10:55:02.008142 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:02.008152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:02.008171 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:55:02.008213 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:02.008218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:02.008432 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:02.008584 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:02.009045 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40072: BlockingRecv error: recv got EOF from 127.0.0.1:40072 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
W20260927 10:55:02.009505 1211 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc)
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:02.010538 1211 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:02.011547 1211 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:55:02.020973 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:02.012389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.1.46.193:45303)
0927 10:55:02.012520 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:02.012523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:02.012573 (+ 50us) server_negotiation.cc:407] Connection header received
0927 10:55:02.012664 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:02.012666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:02.012701 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:02.012765 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:02.013233 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:02.013999 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:02.014950 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:02.015302 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:02.015385 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:02.017722 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:02.017730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:02.017740 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:02.017761 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:02.019489 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:02.019869 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:02.019873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:02.019875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:02.019946 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:02.020154 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:02.020156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:02.020157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:02.020358 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:02.020496 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:02.020641 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 10:55:02.020797 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:55:02.024592 7246 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:02.023634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.1.46.193:45303)
0927 10:55:02.023748 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:02.023752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:02.023839 (+ 87us) server_negotiation.cc:407] Connection header received
0927 10:55:02.023933 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:02.023937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:02.023979 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:02.024045 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:02.024468 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40086: BlockingRecv error: recv got EOF from 127.0.0.1:40086 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 10:55:02.024653 1211 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:02.025561 1211 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:02.026422 1211 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:55:02.034755 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:02.027255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.1.46.193:45303)
0927 10:55:02.027415 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:02.027418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:02.027450 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:55:02.027549 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:02.027551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:02.027581 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:02.027641 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:02.028075 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:02.028637 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:02.029422 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:02.029623 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:02.029693 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:02.031883 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:02.031886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:02.031888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:02.031906 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:02.033214 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:02.033622 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:02.033625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:02.033626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:02.033699 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:02.033945 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:02.033948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:02.033949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:02.034119 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:02.034228 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:02.034417 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 10:55:02.034511 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 10:55:02.037299 7246 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:02.036528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.1.46.193:45303)
0927 10:55:02.036632 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:02.036638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:02.036746 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:55:02.036834 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:02.036836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:02.036865 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:02.036929 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:02.037200 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40106: BlockingRecv error: recv got EOF from 127.0.0.1:40106 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 10:55:02.037462 1211 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790506472751632-1211-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:02.038424 1211 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:02.039305 1211 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[6735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:55:02.046931 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:02.040047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.1.46.193:45303)
0927 10:55:02.040159 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:02.040162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:02.040175 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:02.040281 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:02.040282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:02.040309 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:02.040365 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:02.040776 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:02.041213 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:02.041901 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:02.042085 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:02.042186 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:02.044197 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:02.044201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:02.044204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:02.044234 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:02.045379 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:02.045760 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:02.045765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:02.045767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:02.045841 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:02.046141 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:02.046145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:02.046147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:02.046294 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:02.046418 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:02.046685 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 10:55:02.046783 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:55:02.047832 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6842
I20260927 10:55:02.053261 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6976
I20260927 10:55:02.059293 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 7110
I20260927 10:55:02.064258 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 6748
2026-09-27T10:55:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2416 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[7259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[7259](info): set up 2 sockets
Sep 27 10:55:02 dist-test-slave-cffh krb5kdc[7259](info): commencing operation
krb5kdc: starting...
W20260927 10:55:03.099040 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:55:03 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab.
Entry for principal oryx/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab.
I20260927 10:55:03.217377 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:41443
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab
--principal=oryx/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:03.322691 7272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:03.322940 7272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:03.322980 7272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:03.326886 7272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:03.326937 7272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:03.326956 7272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:03.326972 7272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:03.326987 7272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:03.331977 7272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:41443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab
--principal=oryx/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:03.333178 7272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:03.334473 7272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:03.342975 7280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:03.342972 7277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:03.342972 7278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:03.343088 7272 server_base.cc:1034] running on GCE node
I20260927 10:55:03.343779 7272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:03.344812 7272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:03.346004 7272 hybrid_clock.cc:648] HybridClock initialized: now 1790506503345982 us; error 43 us; skew 500 ppm
Sep 27 10:55:03 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:03.348683 7272 init.cc:377] Logged in from keytab as oryx/127.1.46.254@KRBTEST.COM (short username oryx)
I20260927 10:55:03.349494 7272 webserver.cc:460] Webserver started at http://127.1.46.254:35209/ using document root <none> and password file <none>
I20260927 10:55:03.350023 7272 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:03.350109 7272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:03.350328 7272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:03.352358 7272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "4f2dee0fa9464d7ea72a69731e6112e8"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "248a0527d3ef3338822dc54c35a27b17"
server_key_iv: "b911006aaf0bb8b32f9c76416b851545"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.352914 7272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "4f2dee0fa9464d7ea72a69731e6112e8"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "248a0527d3ef3338822dc54c35a27b17"
server_key_iv: "b911006aaf0bb8b32f9c76416b851545"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.356511 7272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:55:03.358747 7287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:03.359932 7272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:03.360061 7272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "4f2dee0fa9464d7ea72a69731e6112e8"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "248a0527d3ef3338822dc54c35a27b17"
server_key_iv: "b911006aaf0bb8b32f9c76416b851545"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.360162 7272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:03.370776 7272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:03.373284 7272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:03.373507 7272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:03.380555 7272 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:41443
I20260927 10:55:03.380818 7339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:41443 every 8 connection(s)
I20260927 10:55:03.381458 7272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:03.382311 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 7272
I20260927 10:55:03.382447 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:55:03.382750 1211 external_mini_cluster.cc:1428] Setting key 0ea02f0df9c51912a807ef661f88513d
I20260927 10:55:03.384660 7340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:03.391057 7340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: Bootstrap starting.
Sep 27 10:55:03 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:03.393661 7340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:03.394542 7340 log.cc:824] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:03.396551 7340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: No bootstrap required, opened a new log
I20260927 10:55:03.397461 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:03.384022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.1.46.254:41443)
0927 10:55:03.384441 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:03.384461 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:03.384499 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:55:03.385302 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:03.385321 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:03.385703 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:03.386033 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:03.386756 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.388227 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.389063 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.389739 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.389893 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:03.392577 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:03.392583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:03.392597 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:03.392625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:03.394601 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:03.395062 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:03.395067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:03.395073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:03.395236 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:03.395565 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:03.395571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:03.395578 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:03.395945 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:03.396150 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:03.396429 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 10:55:03.396775 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
I20260927 10:55:03.399766 7340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } }
I20260927 10:55:03.400045 7340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:03.400102 7340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2dee0fa9464d7ea72a69731e6112e8, State: Initialized, Role: FOLLOWER
I20260927 10:55:03.400476 7340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } }
I20260927 10:55:03.400589 7340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:03.400626 7340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:03.400684 7340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:03.401520 7340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } }
I20260927 10:55:03.401782 7340 leader_election.cc:302] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f2dee0fa9464d7ea72a69731e6112e8; no voters:
I20260927 10:55:03.402010 7340 leader_election.cc:288] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:03.402164 7345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:03.402441 7345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 1 LEADER]: Becoming Leader. State: Replica: 4f2dee0fa9464d7ea72a69731e6112e8, State: Running, Role: LEADER
I20260927 10:55:03.402766 7345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } }
I20260927 10:55:03.403331 7340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:03.404593 7346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f2dee0fa9464d7ea72a69731e6112e8. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } }
I20260927 10:55:03.404623 7347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } }
I20260927 10:55:03.404712 7346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:03.404731 7347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:03.405068 7354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:03.407805 7354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:03.412901 7354 catalog_manager.cc:1347] Generated new cluster ID: 6fd799a375bc41b7823839243e546672
I20260927 10:55:03.412971 7354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:03.433238 7354 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:03.434208 7354 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:03.442612 7354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: Generated new TSK 0
I20260927 10:55:03.443328 7354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab.
Entry for principal oryx/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab.
I20260927 10:55:03.478634 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41443
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab
--principal=oryx/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:03.585064 7366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:03.585300 7366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:03.585340 7366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:03.589382 7366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:03.589432 7366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:03.589488 7366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:03.594453 7366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab
--principal=oryx/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:03.595410 7366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:03.596381 7366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:03.605741 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:03.605785 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:03.605886 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:03.605785 7366 server_base.cc:1034] running on GCE node
I20260927 10:55:03.606364 7366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:03.607010 7366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:03.608167 7366 hybrid_clock.cc:648] HybridClock initialized: now 1790506503608139 us; error 40 us; skew 500 ppm
Sep 27 10:55:03 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:03.611790 7366 init.cc:377] Logged in from keytab as oryx/127.1.46.193@KRBTEST.COM (short username oryx)
I20260927 10:55:03.612869 7366 webserver.cc:460] Webserver started at http://127.1.46.193:40513/ using document root <none> and password file <none>
I20260927 10:55:03.613644 7366 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:03.613739 7366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:03.614061 7366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:03.617012 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "c52462feee3e45bf8238de90cf56aaa5"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "394991086f4190df7ebf4e02496e3a79"
server_key_iv: "2abf957221dd4993aac5a265164bfc3f"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.617763 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "c52462feee3e45bf8238de90cf56aaa5"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "394991086f4190df7ebf4e02496e3a79"
server_key_iv: "2abf957221dd4993aac5a265164bfc3f"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.623572 7366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 10:55:03.627395 7381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:03.629071 7366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:55:03.629197 7366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "c52462feee3e45bf8238de90cf56aaa5"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "394991086f4190df7ebf4e02496e3a79"
server_key_iv: "2abf957221dd4993aac5a265164bfc3f"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.629320 7366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:03.648734 7366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:03.651868 7366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:03.652076 7366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:03.652643 7366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:03.653463 7366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:03.653532 7366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:03.653600 7366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:03.653656 7366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:03.662333 7366 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:33931
I20260927 10:55:03.662431 7494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:33931 every 8 connection(s)
I20260927 10:55:03.663358 7366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:03.664724 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 7366
I20260927 10:55:03.664860 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:03.665207 1211 external_mini_cluster.cc:1428] Setting key 1363bb22456bbaf55495642863441053
Sep 27 10:55:03 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:55:03.679317 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:03.665249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:56093 (local address 127.1.46.254:41443)
0927 10:55:03.665385 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:03.665390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:03.667013 (+ 1623us) server_negotiation.cc:407] Connection header received
0927 10:55:03.668390 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:03.668393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:03.668444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:03.668521 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:03.669990 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.671210 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.672366 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.672642 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.672756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:03.675534 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:03.675537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:03.675539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:03.675561 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:03.677010 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:03.677620 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:03.677623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:03.677624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:03.677695 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:03.678130 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:03.678134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:03.678137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:03.678311 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:03.678422 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:03.679057 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 10:55:03.679197 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:55:03.680001 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:03.665748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.193:56093)
0927 10:55:03.666859 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:03.666918 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:03.668185 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:03.668684 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:03.668691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:03.669141 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:03.669821 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:03.669840 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.671357 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.671360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:03.672242 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:03.672250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.672789 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.672793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:03.672967 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:03.673519 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:03.673535 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:03.675365 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:03.677135 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:03.677144 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:03.677161 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:03.677519 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:03.677836 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:03.677841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:03.677846 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:03.678026 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:03.678451 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:03.678456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:03.678766 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 10:55:03.678958 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 10:55:03.679193 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.194@KRBTEST.COM" created.
I20260927 10:55:03.681523 7495 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41443
I20260927 10:55:03.681771 7495 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:03.682235 7495 heartbeater.cc:511] Master 127.1.46.254:41443 requested a full tablet report, sending...
I20260927 10:55:03.683701 7304 ts_manager.cc:194] Registered new tserver with Master: c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931)
I20260927 10:55:03.684792 7304 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:56093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab.
Entry for principal oryx/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab.
I20260927 10:55:03.697165 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41443
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab
--principal=oryx/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:03.809316 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:03.809546 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:03.809615 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:03.813619 7500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:03.813671 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:03.813783 7500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:03.818850 7500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab
--principal=oryx/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:03.819770 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:03.820736 7500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:03.829932 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:03.829926 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:03.829939 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:03.830134 7500 server_base.cc:1034] running on GCE node
I20260927 10:55:03.830610 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:03.831214 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:03.832373 7500 hybrid_clock.cc:648] HybridClock initialized: now 1790506503832343 us; error 46 us; skew 500 ppm
Sep 27 10:55:03 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:03.835072 7500 init.cc:377] Logged in from keytab as oryx/127.1.46.194@KRBTEST.COM (short username oryx)
I20260927 10:55:03.835965 7500 webserver.cc:460] Webserver started at http://127.1.46.194:39623/ using document root <none> and password file <none>
I20260927 10:55:03.836486 7500 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:03.836567 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:03.836760 7500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:03.838696 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "916eb4920b88444aa424619c328e0fb4"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "733f8fd4ab35aa30b20cba9df1ef913a"
server_key_iv: "3597eb85dbd2705ae0e660ede2ba271e"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.839275 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "916eb4920b88444aa424619c328e0fb4"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "733f8fd4ab35aa30b20cba9df1ef913a"
server_key_iv: "3597eb85dbd2705ae0e660ede2ba271e"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.842825 7500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:03.845039 7515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:03.846341 7500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:03.846454 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "916eb4920b88444aa424619c328e0fb4"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "733f8fd4ab35aa30b20cba9df1ef913a"
server_key_iv: "3597eb85dbd2705ae0e660ede2ba271e"
server_key_version: "encryptionkey@0"
I20260927 10:55:03.846551 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:03.866101 7500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:03.869082 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:03.869272 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:03.869820 7500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:03.870661 7500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:03.870725 7500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:03.870788 7500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:03.870838 7500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:03.880007 7500 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:37281
I20260927 10:55:03.880028 7628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:37281 every 8 connection(s)
I20260927 10:55:03.880827 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:03.882844 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 7500
I20260927 10:55:03.882963 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:03.883263 1211 external_mini_cluster.cc:1428] Setting key 5915a5fe811f801a982690b7dbc5bb10
Sep 27 10:55:03 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:55:03.895710 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:03.882555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:36853 (local address 127.1.46.254:41443)
0927 10:55:03.882754 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:03.882759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:03.883637 (+ 878us) server_negotiation.cc:407] Connection header received
0927 10:55:03.885603 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:03.885606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:03.885653 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:03.885795 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:03.887205 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.887939 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.889027 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.889256 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.889365 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:03.892150 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:03.892156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:03.892159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:03.892187 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:03.893796 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:03.894259 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:03.894262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:03.894263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:03.894333 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:03.894669 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:03.894671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:03.894673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:03.894833 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:03.894905 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:03.895346 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 10:55:03.895485 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.195@KRBTEST.COM" created.
I20260927 10:55:03.896539 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:03.882975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:36853)
0927 10:55:03.883471 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:03.883524 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:03.885331 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:03.885980 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:03.885992 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:03.886491 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:03.887053 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:03.887067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.888045 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.888047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:03.888895 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:03.888903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:03.889396 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:03.889400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:03.889586 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:03.890259 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:03.890276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:03.891981 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:03.893902 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:03.893907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:03.893917 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:03.894158 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:03.894409 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:03.894411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:03.894413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:03.894544 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:03.894920 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:03.894925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:03.895143 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 10:55:03.895286 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 10:55:03.895586 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260927 10:55:03.897718 7629 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41443
I20260927 10:55:03.897962 7629 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:03.898424 7629 heartbeater.cc:511] Master 127.1.46.254:41443 requested a full tablet report, sending...
I20260927 10:55:03.899586 7304 ts_manager.cc:194] Registered new tserver with Master: 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281)
I20260927 10:55:03.900161 7304 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:36853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab.
Entry for principal oryx/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab.
I20260927 10:55:03.912842 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41443
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab
--principal=oryx/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:04.019014 7634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:04.019266 7634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:04.019336 7634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:04.023245 7634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:04.023319 7634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:04.023401 7634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:04.028402 7634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab
--principal=oryx/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:04.029368 7634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:04.030333 7634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:04.039367 7639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.039425 7642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.039409 7640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.039682 7634 server_base.cc:1034] running on GCE node
I20260927 10:55:04.040040 7634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:04.040607 7634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:04.041805 7634 hybrid_clock.cc:648] HybridClock initialized: now 1790506504041792 us; error 30 us; skew 500 ppm
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:04.044409 7634 init.cc:377] Logged in from keytab as oryx/127.1.46.195@KRBTEST.COM (short username oryx)
I20260927 10:55:04.045228 7634 webserver.cc:460] Webserver started at http://127.1.46.195:36047/ using document root <none> and password file <none>
I20260927 10:55:04.045749 7634 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.045837 7634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.046053 7634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:04.048126 7634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "53a1dac39e57495fad5fc4a00e2e1eed"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "4573b4e6dedba5c657506d967fc0bc6e"
server_key_iv: "db13b92b84c8a3b82e20f527bd906819"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.048662 7634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "53a1dac39e57495fad5fc4a00e2e1eed"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "4573b4e6dedba5c657506d967fc0bc6e"
server_key_iv: "db13b92b84c8a3b82e20f527bd906819"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.052181 7634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:55:04.054404 7649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.055600 7634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:04.055727 7634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "53a1dac39e57495fad5fc4a00e2e1eed"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "4573b4e6dedba5c657506d967fc0bc6e"
server_key_iv: "db13b92b84c8a3b82e20f527bd906819"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.055831 7634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.071305 7634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:04.074046 7634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:04.074244 7634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:04.074837 7634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:04.075605 7634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:04.075672 7634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.075742 7634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:04.075785 7634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.084273 7634 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39857
I20260927 10:55:04.084298 7762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39857 every 8 connection(s)
I20260927 10:55:04.085146 7634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:04.088333 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 7634
I20260927 10:55:04.088460 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:04.088833 1211 external_mini_cluster.cc:1428] Setting key 6f599eccf4f18fec7d7a47bc55ea9644
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:04.098526 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.087035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:37917 (local address 127.1.46.254:41443)
0927 10:55:04.087175 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.087179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.088005 (+ 826us) server_negotiation.cc:407] Connection header received
0927 10:55:04.089661 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.089664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.089719 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.089795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.091343 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.091858 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.092978 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.093176 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.093248 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.095593 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.095596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.095598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.095617 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.096873 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.097319 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.097321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.097322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.097388 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.097675 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.097677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.097678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.097825 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.097897 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.098319 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.098410 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:55:04.099350 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.087434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.195:37917)
0927 10:55:04.087865 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.087905 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.089421 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.090005 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.090014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.090630 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.091223 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.091237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.091973 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.091976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.092877 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.092885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.093305 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.093308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.093470 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.093995 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.094009 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.095462 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.096980 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.096984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.096995 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.097239 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.097471 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.097473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.097474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.097598 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.097908 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.097913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.098115 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.098269 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.098531 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260927 10:55:04.100473 7763 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41443
I20260927 10:55:04.100744 7763 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:04.101254 7763 heartbeater.cc:511] Master 127.1.46.254:41443 requested a full tablet report, sending...
I20260927 10:55:04.102336 7304 ts_manager.cc:194] Registered new tserver with Master: 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:04.103096 7304 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:37917
I20260927 10:55:04.114320 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.253.keytab.
Entry for principal oryx/127.1.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.253.keytab.
I20260927 10:55:04.142189 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:38199
--webserver_interface=127.1.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.253.keytab
--principal=oryx/127.1.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:41443,127.1.46.253:38199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:04.251019 7768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:04.251263 7768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:04.251302 7768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:04.255165 7768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:04.255216 7768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:04.255235 7768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:04.255251 7768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:04.255267 7768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:04.260443 7768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:41443,127.1.46.253:38199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.253.keytab
--principal=oryx/127.1.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:38199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
W20260927 10:55:04.260620 7768 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.46.254:41443,127.1.46.253:38199'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 10:55:04.261394 7768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:04.262307 7768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:04.270510 7774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.270534 7773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.270661 7768 server_base.cc:1034] running on GCE node
W20260927 10:55:04.270534 7776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.271363 7768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:04.272508 7768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:04.273690 7768 hybrid_clock.cc:648] HybridClock initialized: now 1790506504273676 us; error 29 us; skew 500 ppm
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:04.276353 7768 init.cc:377] Logged in from keytab as oryx/127.1.46.253@KRBTEST.COM (short username oryx)
I20260927 10:55:04.277184 7768 webserver.cc:460] Webserver started at http://127.1.46.253:35107/ using document root <none> and password file <none>
I20260927 10:55:04.277936 7768 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.278050 7768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.278302 7768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:04.280494 7768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/instance:
uuid: "3f45cab475a749b48659c4108a524459"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "b0aefe06dc2aaf3ffc73d71e37a311f2"
server_key_iv: "5080f1bc95adb5945b3e4051b7ecd228"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.281049 7768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal/instance:
uuid: "3f45cab475a749b48659c4108a524459"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "b0aefe06dc2aaf3ffc73d71e37a311f2"
server_key_iv: "5080f1bc95adb5945b3e4051b7ecd228"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.284796 7768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:55:04.287213 7783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.288532 7768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:04.288684 7768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
uuid: "3f45cab475a749b48659c4108a524459"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "b0aefe06dc2aaf3ffc73d71e37a311f2"
server_key_iv: "5080f1bc95adb5945b3e4051b7ecd228"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.288810 7768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.314051 7768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:04.317215 7768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:04.317444 7768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:04.324479 7768 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.253:38199
I20260927 10:55:04.324759 7835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.253:38199 every 8 connection(s)
I20260927 10:55:04.325459 7768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
I20260927 10:55:04.328287 7836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:04.328526 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 7768
I20260927 10:55:04.332836 7836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.46.253@KRBTEST.COM
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.253@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:04.344935 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.330043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.1.46.253:38199)
0927 10:55:04.330485 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.330500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.330534 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:55:04.331483 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.331502 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.331823 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.332179 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.332991 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.334033 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.335041 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.335735 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.336316 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.339700 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.339706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.339716 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.339738 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.341575 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.342061 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.342068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.342075 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.342221 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.342557 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.342561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.342563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.343152 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.343316 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.343705 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.344054 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260927 10:55:04.345099 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.334607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.1.46.254:41443)
0927 10:55:04.334727 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.334732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.335050 (+ 318us) server_negotiation.cc:407] Connection header received
0927 10:55:04.335226 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.335229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.335278 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.335359 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.336284 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.337175 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.338329 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.338681 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.338789 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.341149 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.341153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.341156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.341191 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.342925 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.343565 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.343568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.343570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.343668 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.343999 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.344003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.344006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.344268 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.344385 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.344831 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.344950 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:55:04.345238 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.334528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60116)
0927 10:55:04.334923 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.334972 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.335136 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.335411 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.335415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.335956 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.336134 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.336145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.337342 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.337346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.338198 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.338207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.338847 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.338851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.338970 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.339504 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.339524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.340993 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.343070 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.343077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.343082 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.343452 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.343785 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.343788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.343790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.343889 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.344438 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.344445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.344542 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.344750 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.344912 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":171,"threads_started":1}
I20260927 10:55:04.346999 7836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.253@KRBTEST.COM for oryx/127.1.46.253@KRBTEST.COM
I20260927 10:55:04.355309 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.347494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.0.0.1:40248)
0927 10:55:04.347612 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.347629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.347706 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.347926 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.347929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.348062 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.348133 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.348137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.349194 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.349197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.349728 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.349736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.350244 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.350247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.350307 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.350904 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.350915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.352181 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.353617 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.353619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.353620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.353789 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.354055 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.354061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.354065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.354191 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.354695 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.354700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.354792 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.354905 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.355070 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:55:04.355573 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.347652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.1.46.253:38199)
0927 10:55:04.347757 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.347760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.347771 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:55:04.347797 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.347799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.347829 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.347892 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.348224 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.349081 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.349840 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.350121 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.350203 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.352298 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.352301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.352303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.352322 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.353518 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.353867 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.353869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.353870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.353932 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.354272 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.354279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.354300 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.354531 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.354657 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.355015 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.355167 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:55:04.359247 7836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Bootstrap starting.
I20260927 10:55:04.362236 7836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:04.363145 7836 log.cc:824] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:04.365387 7836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: No bootstrap required, opened a new log
I20260927 10:55:04.368368 7836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } }
I20260927 10:55:04.368566 7836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:04.368600 7836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f45cab475a749b48659c4108a524459, State: Initialized, Role: FOLLOWER
I20260927 10:55:04.369020 7836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } }
I20260927 10:55:04.369421 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } } }
I20260927 10:55:04.369565 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:04.370222 7836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:55:04.374902 7853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:55:04.374984 7853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:04.377333 7768 master_runner.cc:428] Detected that this master 3f45cab475a749b48659c4108a524459 is joining an existing cluster
I20260927 10:55:04.377406 7768 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.46.253:38199
I20260927 10:55:04.388473 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.379835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.1.46.254:41443)
0927 10:55:04.379970 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.379974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.380381 (+ 407us) server_negotiation.cc:407] Connection header received
0927 10:55:04.380579 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.380584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.380636 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.380677 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.381205 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.381972 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.383316 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.383663 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.383728 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.385016 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.385019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.385022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.385045 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.386694 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.387237 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.387241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.387242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.387321 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.387621 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.387624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.387626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.387824 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.387887 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.388196 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.388310 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:55:04.388983 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.379772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60120)
0927 10:55:04.380302 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.380315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.380473 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.380783 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.380785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.380950 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.381091 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.381096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.382100 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.382103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.383192 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.383202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.383863 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.383866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.383966 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.384413 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.384426 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.384877 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.386812 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.386815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.386817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.387124 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.387447 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.387450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.387452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.387532 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.387952 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.387955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.388036 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.388655 (+ 619us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.388790 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":91,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260927 10:55:04.389268 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.380176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.1.46.253:38199)
0927 10:55:04.380294 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.380298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.380314 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:55:04.380357 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.380362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.380410 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.380492 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.381032 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.381581 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.382760 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.383099 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.383200 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.384981 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.384985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.384988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.385008 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.387289 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.387743 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.387746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.387748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.387813 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.388081 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.388084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.388086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.388275 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.388357 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.389022 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.389124 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":49}
I20260927 10:55:04.390828 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.379772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.0.0.1:40256)
0927 10:55:04.380075 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.380089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.380183 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.380535 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.380539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.380772 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.380902 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.380909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.381688 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.381694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.382641 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.382652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.383349 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.383353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.383456 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.384013 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.384029 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.384853 (+ 824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.387420 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.387422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.387425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.387646 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.387908 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.387911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.387913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.387989 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.388449 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.388452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.388858 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.390488 (+ 1630us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.390648 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":25,"thread_start_us":124,"threads_started":1}
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.254@KRBTEST.COM for oryx/127.1.46.253@KRBTEST.COM
I20260927 10:55:04.404352 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.393827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.0.0.1:40270)
0927 10:55:04.394829 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.394877 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.395000 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.395303 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.395309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.395732 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.395849 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.395856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.396845 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.396852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.397702 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.397712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.398275 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.398278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.398352 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.398915 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.398927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.400470 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.402447 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.402460 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.402469 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.402835 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.403127 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.403131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.403134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.403236 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.403688 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.403697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.403819 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.403981 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.404183 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":90,"threads_started":1}
I20260927 10:55:04.404343 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.393876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.1.46.253:38199)
0927 10:55:04.393976 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.393980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.394954 (+ 974us) server_negotiation.cc:407] Connection header received
0927 10:55:04.395129 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.395132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.395182 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.395263 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.395967 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.396723 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.397791 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.398070 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.398167 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.400614 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.400618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.400620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.400646 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.402323 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.402939 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.402942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.402944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.403023 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.403341 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.403344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.403346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.403552 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.403646 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.404055 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.404182 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:55:04.405934 7304 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.46.253:38199: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3f45cab475a749b48659c4108a524459"
member_type: NON_VOTER
last_known_addr {
host: "127.1.46.253"
port: 38199
}
attrs {
promote: true
}
}
dest_uuid: "4f2dee0fa9464d7ea72a69731e6112e8"
cas_config_opid_index: -1
I20260927 10:55:04.406728 7304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.1.46.252@KRBTEST.COM; defaulting to no policy
I20260927 10:55:04.408692 7863 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3f45cab475a749b48659c4108a524459 (127.1.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } }
I20260927 10:55:04.409591 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.252@KRBTEST.COM" created.
I20260927 10:55:04.411608 7863 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.46.253:38199
I20260927 10:55:04.411710 7810 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Refusing update from remote peer 4f2dee0fa9464d7ea72a69731e6112e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 10:55:04.411818 7863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } }
I20260927 10:55:04.411921 7863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:04.412611 7768 master.cc:537] Master@127.1.46.253:38199 shutting down...
I20260927 10:55:04.412633 7863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:55:04.415264 7288 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 10:55:04.416755 7288 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 -> Peer 3f45cab475a749b48659c4108a524459 (127.1.46.253:38199): Couldn't send request to peer 3f45cab475a749b48659c4108a524459. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:55:04.417634 7768 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:04.417934 7768 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:04.418058 7768 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: stopping tablet replica
I20260927 10:55:04.422392 7768 master.cc:552] Master@127.1.46.253:38199 shutdown complete.
I20260927 10:55:04.422461 7768 master_runner.cc:305] Clearing existing system tablet
I20260927 10:55:04.422729 7768 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.422802 7768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.423856 7768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:04.424800 7869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.425084 7768 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.425203 7768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:04.425256 7768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
uuid: "3f45cab475a749b48659c4108a524459"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "b0aefe06dc2aaf3ffc73d71e37a311f2"
server_key_iv: "5080f1bc95adb5945b3e4051b7ecd228"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.427251 7768 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:04.430531 7768 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:55:04.430706 7768 log.cc:1196] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 10:55:04.431051 7768 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Deleting consensus metadata
I20260927 10:55:04.431217 7768 master_runner.cc:315] Copying system tablet from 127.1.46.254:41443
I20260927 10:55:04.432500 7768 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: tablet copy: Beginning tablet copy session from remote peer at address 127.1.46.254:41443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.252.keytab.
Entry for principal oryx/127.1.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.252.keytab.
I20260927 10:55:04.433502 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:35733
--webserver_interface=127.1.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.252.keytab
--principal=oryx/127.1.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:04.442629 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.433473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.1.46.254:41443)
0927 10:55:04.433636 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.433643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.435112 (+ 1469us) server_negotiation.cc:407] Connection header received
0927 10:55:04.435277 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.435281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.435335 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.435434 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.435939 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.436523 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.437485 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.437737 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.437859 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.439481 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.439485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.439489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.439515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.441005 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.441427 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.441432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.441434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.441512 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.441804 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.441810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.441812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.441987 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.442093 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.442344 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.442460 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:55:04.442644 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.433072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60130)
0927 10:55:04.435003 (+ 1931us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.435016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.435163 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.435550 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.435553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.435718 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.435816 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.435821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.436683 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.436689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.437363 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.437373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.437860 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.437863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.437965 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.438975 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.438988 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.439306 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.441128 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.441132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.441135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.441327 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.441613 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.441616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.441619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.441717 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.442084 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.442087 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.442186 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.442291 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.442424 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843,"thread_start_us":90,"threads_started":1}
I20260927 10:55:04.443550 7324 tablet_copy_service.cc:140] P 4f2dee0fa9464d7ea72a69731e6112e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3f45cab475a749b48659c4108a524459 ({username='oryx', principal='oryx/127.1.46.253@KRBTEST.COM'} at 127.0.0.1:60130)
I20260927 10:55:04.443727 7324 tablet_copy_service.cc:161] P 4f2dee0fa9464d7ea72a69731e6112e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3f45cab475a749b48659c4108a524459 at {username='oryx', principal='oryx/127.1.46.253@KRBTEST.COM'} at 127.0.0.1:60130: session id = 3f45cab475a749b48659c4108a524459-00000000000000000000000000000000
I20260927 10:55:04.445143 7324 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:55:04.446729 7768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:55:04.448719 7768 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: tablet copy: Starting download of 0 data blocks...
I20260927 10:55:04.448916 7768 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: tablet copy: Starting download of 1 WAL segments...
I20260927 10:55:04.450537 7768 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:55:04.452056 7324 tablet_copy_service.cc:342] P 4f2dee0fa9464d7ea72a69731e6112e8: Request end of tablet copy session 3f45cab475a749b48659c4108a524459-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.46.253@KRBTEST.COM'} at 127.0.0.1:60130
I20260927 10:55:04.452178 7324 tablet_copy_service.cc:433] P 4f2dee0fa9464d7ea72a69731e6112e8: ending tablet copy session 3f45cab475a749b48659c4108a524459-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3f45cab475a749b48659c4108a524459
I20260927 10:55:04.459173 7768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:04.460852 7880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.460947 7877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.461149 7768 server_base.cc:1034] running on GCE node
W20260927 10:55:04.461385 7878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.461615 7768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:04.461979 7768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:04.463122 7768 hybrid_clock.cc:648] HybridClock initialized: now 1790506504463119 us; error 22 us; skew 500 ppm
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:04.464766 7768 init.cc:377] Logged in from keytab as oryx/127.1.46.253@KRBTEST.COM (short username oryx)
I20260927 10:55:04.465235 7768 webserver.cc:460] Webserver started at http://127.1.46.253:38347/ using document root <none> and password file <none>
I20260927 10:55:04.465420 7768 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.465480 7768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.466549 7768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:04.467512 7887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.467746 7768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:55:04.467839 7768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
uuid: "3f45cab475a749b48659c4108a524459"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "b0aefe06dc2aaf3ffc73d71e37a311f2"
server_key_iv: "5080f1bc95adb5945b3e4051b7ecd228"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.467938 7768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.484995 7768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:04.487162 7768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:04.492169 7768 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.253:38199
I20260927 10:55:04.492765 7768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
I20260927 10:55:04.494230 7940 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:55:04.495137 7940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Bootstrap starting.
I20260927 10:55:04.502718 7939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.253:38199 every 8 connection(s)
I20260927 10:55:04.503059 7940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:04.503355 7940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Bootstrap complete.
I20260927 10:55:04.504184 7940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } }
I20260927 10:55:04.504321 7940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f45cab475a749b48659c4108a524459, State: Initialized, Role: LEARNER
I20260927 10:55:04.504439 7940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } }
I20260927 10:55:04.504819 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } }
I20260927 10:55:04.504992 7940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:04.505015 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:55:04.507800 7954 catalog_manager.cc:1259] Loaded cluster ID: 6fd799a375bc41b7823839243e546672
I20260927 10:55:04.507918 7954 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: loading cluster ID for follower catalog manager: success
I20260927 10:55:04.509584 7954 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: acquiring CA information for follower catalog manager: success
I20260927 10:55:04.510723 7954 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:55:04.583031 7876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:04.583333 7876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:04.583415 7876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:04.588196 7876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:04.588279 7876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:04.588317 7876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:04.588354 7876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:04.588388 7876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:04.593819 7876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.252.keytab
--principal=oryx/127.1.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:04.595013 7876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:04.596043 7876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:04.604763 7959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.604770 7962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.604712 7960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.605073 7876 server_base.cc:1034] running on GCE node
I20260927 10:55:04.605726 7876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:04.607003 7876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:04.608211 7876 hybrid_clock.cc:648] HybridClock initialized: now 1790506504608192 us; error 26 us; skew 500 ppm
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:04.611092 7876 init.cc:377] Logged in from keytab as oryx/127.1.46.252@KRBTEST.COM (short username oryx)
I20260927 10:55:04.611944 7876 webserver.cc:460] Webserver started at http://127.1.46.252:38735/ using document root <none> and password file <none>
I20260927 10:55:04.612504 7876 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.612591 7876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.612818 7876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:04.614950 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/instance:
uuid: "2604f28e264e450c8cab0f21189cd94d"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "3511eec709ec41f3f6546d87210d1c32"
server_key_iv: "c9b9d4654f7ffdd98fa841329ad276b4"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.615496 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal/instance:
uuid: "2604f28e264e450c8cab0f21189cd94d"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "3511eec709ec41f3f6546d87210d1c32"
server_key_iv: "c9b9d4654f7ffdd98fa841329ad276b4"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.619138 7876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:04.621598 7969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.622838 7876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:04.622984 7876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
uuid: "2604f28e264e450c8cab0f21189cd94d"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "3511eec709ec41f3f6546d87210d1c32"
server_key_iv: "c9b9d4654f7ffdd98fa841329ad276b4"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.623091 7876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.657063 7876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:04.660562 7876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:04.660761 7876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:04.669549 7876 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.252:35733
I20260927 10:55:04.669871 8021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.252:35733 every 8 connection(s)
I20260927 10:55:04.670522 7876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
I20260927 10:55:04.671118 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 7876
I20260927 10:55:04.673652 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:04.677220 8022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.46.252@KRBTEST.COM
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.252@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:04.686610 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.672717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.1.46.252:35733)
0927 10:55:04.673138 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.673152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.673187 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:55:04.673901 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.673918 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.674237 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.674598 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.675403 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.676384 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.677241 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.677774 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.677941 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.681145 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.681152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.681170 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.681196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.683295 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.683733 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.683743 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.683751 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.683900 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.684209 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.684212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.684213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.684693 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.684845 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.685205 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.685544 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260927 10:55:04.687083 7495 heartbeater.cc:503] Master 127.1.46.254:41443 was elected leader, sending a full tablet report...
I20260927 10:55:04.688755 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.678997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60136)
0927 10:55:04.679369 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.679407 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.679542 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.679844 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.679849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.680192 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.680305 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.680310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.681157 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.681161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.681995 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.682004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.682504 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.682511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.682669 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.683186 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.683207 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.684797 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.686867 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.686875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.686879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.687187 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.687553 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.687558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.687560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.687657 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.688092 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.688098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.688213 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.688368 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.688545 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":111,"threads_started":1}
I20260927 10:55:04.688849 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.679155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.1.46.254:41443)
0927 10:55:04.679281 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.679286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.679514 (+ 228us) server_negotiation.cc:407] Connection header received
0927 10:55:04.679653 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.679657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.679710 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.679798 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.680432 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.681014 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.682120 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.682384 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.682470 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.684968 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.684971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.684973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.684999 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.686716 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.687318 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.687322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.687324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.687423 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.687764 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.687767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.687769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.687974 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.688076 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.688565 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.688696 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:55:04.689826 8022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.252@KRBTEST.COM for oryx/127.1.46.253@KRBTEST.COM
I20260927 10:55:04.698261 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.690272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.0.0.1:40274)
0927 10:55:04.690414 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.690430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.690511 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.690961 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.690963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.691091 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.691180 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.691185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.692080 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.692089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.693048 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.693059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.693618 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.693621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.693691 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.694093 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.694103 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.695372 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.697111 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.697113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.697115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.697285 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.697512 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.697514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.697515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.697581 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.697879 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.697881 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.697947 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.698053 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.698146 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 10:55:04.698290 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.690437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.1.46.253:38199)
0927 10:55:04.690750 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.690754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.690766 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:04.690801 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.690803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.690841 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.690918 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.691280 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.691958 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.693196 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.693520 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.693610 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.695498 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.695501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.695503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.695525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.697010 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.697365 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.697368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.697369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.697436 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.697653 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.697655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.697656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.697806 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.697876 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.698084 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.698172 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":133,"threads_started":1}
I20260927 10:55:04.699116 8022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.252@KRBTEST.COM for oryx/127.1.46.252@KRBTEST.COM
I20260927 10:55:04.707075 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.699612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.0.0.1:58722)
0927 10:55:04.699697 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.699713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.699784 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.699948 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.699950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.700067 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.700127 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.700131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.701170 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.701174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.701684 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.701692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.702107 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.702110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.702164 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.702518 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.702528 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.703920 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.705739 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.705745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.705749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.705937 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.706214 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.706217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.706220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.706293 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.706660 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.706662 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.706747 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.706869 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.706973 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 10:55:04.707191 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.699659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.1.46.252:35733)
0927 10:55:04.699777 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.699780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.699792 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:04.699839 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.699841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.699871 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.699941 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.700215 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.701034 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.701788 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.702015 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.702093 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.704140 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.704144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.704146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.704167 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.705644 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.706026 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.706032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.706035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.706104 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.706380 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.706384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.706386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.706534 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.706660 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.706935 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.707027 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:55:04.710287 8022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Bootstrap starting.
I20260927 10:55:04.712554 8022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:04.713265 8022 log.cc:824] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:04.715157 8022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: No bootstrap required, opened a new log
I20260927 10:55:04.717974 8022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } }
I20260927 10:55:04.718159 8022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:04.718192 8022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2604f28e264e450c8cab0f21189cd94d, State: Initialized, Role: FOLLOWER
I20260927 10:55:04.718677 8022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } }
I20260927 10:55:04.719130 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } } }
I20260927 10:55:04.719269 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:04.719813 8022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:55:04.724483 8039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:55:04.724571 8039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:04.727907 7876 master_runner.cc:428] Detected that this master 2604f28e264e450c8cab0f21189cd94d is joining an existing cluster
I20260927 10:55:04.727977 7876 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.46.252:35733
I20260927 10:55:04.740335 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 7366
I20260927 10:55:04.747138 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:33931
--local_ip_for_outbound_sockets=127.1.46.193
--tserver_master_addrs=127.1.46.254:41443
--webserver_port=40513
--webserver_interface=127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab
--principal=oryx/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:04.759639 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.730371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.0.0.1:58734)
0927 10:55:04.731076 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.731088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.731185 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.731381 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.731384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.731610 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.731698 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.731703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.732405 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.732408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.733000 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.733007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.733437 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.733440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.733540 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.734482 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.734499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.734951 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.757483 (+ 22532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.757487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.757489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.757743 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.758020 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.758022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.758025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.758104 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.759026 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.759029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.759132 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.759252 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.759414 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260927 10:55:04.760280 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.730414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.1.46.252:35733)
0927 10:55:04.730662 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.730665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.731139 (+ 474us) server_negotiation.cc:407] Connection header received
0927 10:55:04.731253 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.731257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.731294 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.731370 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.731814 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.732307 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.733114 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.733319 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.733432 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.735082 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.735086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.735088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.735112 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.757084 (+ 21972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.757848 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.757851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.757853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.757925 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.758200 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.758203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.758206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.758859 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.758954 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.759988 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.760119 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":148}
I20260927 10:55:04.761680 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.730755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60140)
0927 10:55:04.738608 (+ 7853us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.738631 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.738741 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.739037 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.739039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.739236 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.739365 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.739372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.740484 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.740488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.741239 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.741249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.742045 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.742051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.742172 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.742675 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.742690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.756665 (+ 13975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.759408 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.759411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.759413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.759612 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.760186 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.760189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.760191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.760266 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.761217 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.761219 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.761305 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.761406 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.761535 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7750,"thread_start_us":84,"threads_started":1}
I20260927 10:55:04.762375 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.730295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.1.46.254:41443)
0927 10:55:04.730539 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.730544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.738710 (+ 8166us) server_negotiation.cc:407] Connection header received
0927 10:55:04.738860 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.738862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.738916 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.739001 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.739523 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.740350 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.741360 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.741775 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.741890 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.756862 (+ 14972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.756868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.756871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.756917 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.759297 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.759993 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.759997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.759999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.760082 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.760861 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.760864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.760866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.761081 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.761172 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.762094 (+ 922us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.762225 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260927 10:55:04.762727 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.730248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.0.0.1:40282)
0927 10:55:04.730991 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.731005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.731110 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.731444 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.731447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.731701 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.731856 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.731864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.732943 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.732947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.733848 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.733858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.734614 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.734618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.734709 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.735644 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.735657 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.755358 (+ 19701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.758357 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.758361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.758364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.759871 (+ 1507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.760680 (+ 809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.760683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.760687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.760767 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.762226 (+ 1459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.762229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.762310 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.762409 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.762546 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":107,"threads_started":1}
I20260927 10:55:04.763300 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.730664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.1.46.253:38199)
0927 10:55:04.730809 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.730816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.731094 (+ 278us) server_negotiation.cc:407] Connection header received
0927 10:55:04.731207 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.731210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.731259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.731332 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.732007 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.732781 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.733999 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.734354 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.734431 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.755601 (+ 21170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.755609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.755615 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.755660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.757877 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.760487 (+ 2610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.760491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.760492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.760580 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.761843 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.761849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.761854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.762090 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.762180 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.763009 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.763133 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506503, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.254@KRBTEST.COM for oryx/127.1.46.252@KRBTEST.COM
I20260927 10:55:04.773881 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.766208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.0.0.1:58736)
0927 10:55:04.766368 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.766382 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.766462 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.766843 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.766846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.767027 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.767105 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.767111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.767909 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.767913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.768506 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.768515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.769013 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.769017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.769115 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.769552 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.769565 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.770787 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.772443 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.772448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.772451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.772651 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.772921 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.772926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.772929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.773013 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.773361 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.773364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.773471 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.773576 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.773714 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 10:55:04.773875 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.766247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.1.46.252:35733)
0927 10:55:04.766343 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.766346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.766423 (+ 77us) server_negotiation.cc:407] Connection header received
0927 10:55:04.766695 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.766698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.766741 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.766828 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.767209 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.767798 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.768621 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.768892 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.769012 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.770929 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.770932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.770933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.770952 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.772336 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.772746 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.772752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.772755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.772826 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.773108 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.773112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.773114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.773269 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.773369 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.773622 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.773720 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:55:04.775055 7304 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.46.252:35733: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2604f28e264e450c8cab0f21189cd94d"
member_type: NON_VOTER
last_known_addr {
host: "127.1.46.252"
port: 35733
}
attrs {
promote: true
}
}
dest_uuid: "4f2dee0fa9464d7ea72a69731e6112e8"
cas_config_opid_index: 5
I20260927 10:55:04.775609 7304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } }
I20260927 10:55:04.776676 7864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2604f28e264e450c8cab0f21189cd94d (127.1.46.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.778250 7864 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.46.252:35733
I20260927 10:55:04.778422 7864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.778504 7864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:04.779498 7996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:04.780925 7996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 FOLLOWER]: Refusing update from remote peer 4f2dee0fa9464d7ea72a69731e6112e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:55:04.781509 7863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:04.781908 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.777051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.1.46.253:38199)
0927 10:55:04.777207 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.777212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.777451 (+ 239us) server_negotiation.cc:407] Connection header received
0927 10:55:04.777501 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.777504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.777559 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.777638 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:04.778002 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.778872 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.780366 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.781051 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.781145 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.781324 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.781684 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.781758 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 10:55:04.782382 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.776799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.0.0.1:40290)
0927 10:55:04.777288 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.777301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.777375 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.777750 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.777752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.777767 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:04.777859 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.777864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.779008 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.779011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.780231 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.780241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.781385 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.781388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.781480 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.781525 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.781600 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.782248 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420}
I20260927 10:55:04.783043 7914 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 LEARNER]: Refusing update from remote peer 4f2dee0fa9464d7ea72a69731e6112e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:55:04.783624 7863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 10:55:04.785367 7914 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2604f28e264e450c8cab0f21189cd94d (127.1.46.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.786693 7996 raft_consensus.cc:858] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } }
W20260927 10:55:04.786791 7996 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } }
I20260927 10:55:04.786866 7876 master.cc:537] Master@127.1.46.252:35733 shutting down...
I20260927 10:55:04.788699 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f2dee0fa9464d7ea72a69731e6112e8. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.788817 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:55:04.788910 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.789000 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:55:04.789034 7996 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3f45cab475a749b48659c4108a524459 (127.1.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } }
I20260927 10:55:04.790477 7863 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: attempting to promote NON_VOTER 3f45cab475a749b48659c4108a524459 to VOTER
I20260927 10:55:04.791002 7863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } }
I20260927 10:55:04.791658 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f2dee0fa9464d7ea72a69731e6112e8. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } }
I20260927 10:55:04.791769 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 10:55:04.791855 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } }
I20260927 10:55:04.791918 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 10:55:04.791986 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: NON_VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: true } } }
I20260927 10:55:04.792053 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 10:55:04.792549 7914 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 LEARNER]: Refusing update from remote peer 4f2dee0fa9464d7ea72a69731e6112e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260927 10:55:04.793200 7289 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 -> Peer 2604f28e264e450c8cab0f21189cd94d (127.1.46.252:35733): Couldn't send request to peer 2604f28e264e450c8cab0f21189cd94d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:55:04.793254 7863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 10:55:04.795009 7864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3f45cab475a749b48659c4108a524459 (127.1.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.795603 7914 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3f45cab475a749b48659c4108a524459 (127.1.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.796576 7864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.796676 7864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:04.796814 7864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.796911 7864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:04.797451 8052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.797595 8052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:04.800055 7876 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 10:55:04.800272 7876 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 10:55:04.800350 7876 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: stopping tablet replica
I20260927 10:55:04.817643 7876 master.cc:552] Master@127.1.46.252:35733 shutdown complete.
I20260927 10:55:04.817734 7876 master_runner.cc:305] Clearing existing system tablet
I20260927 10:55:04.817968 7876 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.818061 7876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.819249 7876 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 10:55:04.820192 8061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.820459 7876 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.820616 7876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:04.820652 7876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
uuid: "2604f28e264e450c8cab0f21189cd94d"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "3511eec709ec41f3f6546d87210d1c32"
server_key_iv: "c9b9d4654f7ffdd98fa841329ad276b4"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.822252 7876 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:04.824203 7876 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:55:04.824308 7876 log.cc:1196] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 10:55:04.824570 7876 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Deleting consensus metadata
I20260927 10:55:04.824765 7876 master_runner.cc:315] Copying system tablet from 127.1.46.254:41443
I20260927 10:55:04.825917 7876 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: tablet copy: Beginning tablet copy session from remote peer at address 127.1.46.254:41443
I20260927 10:55:04.834301 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.826587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.1.46.254:41443)
0927 10:55:04.826723 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.826728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.826938 (+ 210us) server_negotiation.cc:407] Connection header received
0927 10:55:04.827086 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.827090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.827147 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.827231 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.827745 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.828362 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.829663 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.829902 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.830020 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.831151 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:04.831156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:04.831159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:04.831185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:04.832656 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.833075 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.833080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.833083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.833177 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:04.833456 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:04.833461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:04.833464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:04.833644 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:04.833749 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:04.834031 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 10:55:04.834138 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:55:04.834316 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.826483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60146)
0927 10:55:04.826842 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.826854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.826987 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.827259 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.827262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.827519 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.827621 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.827627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.828473 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.828477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.829529 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.829538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.830017 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.830020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.830114 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.830659 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.830675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:04.831012 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:04.832780 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.832784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.832787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.832980 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.833277 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:04.833281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:04.833284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:04.833369 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:04.833739 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:04.833742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:04.833860 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 10:55:04.833963 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:55:04.834106 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260927 10:55:04.835157 7324 tablet_copy_service.cc:140] P 4f2dee0fa9464d7ea72a69731e6112e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2604f28e264e450c8cab0f21189cd94d ({username='oryx', principal='oryx/127.1.46.252@KRBTEST.COM'} at 127.0.0.1:60146)
I20260927 10:55:04.835268 7324 tablet_copy_service.cc:161] P 4f2dee0fa9464d7ea72a69731e6112e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2604f28e264e450c8cab0f21189cd94d at {username='oryx', principal='oryx/127.1.46.252@KRBTEST.COM'} at 127.0.0.1:60146: session id = 2604f28e264e450c8cab0f21189cd94d-00000000000000000000000000000000
I20260927 10:55:04.836087 7324 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:55:04.837255 7876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:55:04.839056 7876 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: tablet copy: Starting download of 0 data blocks...
I20260927 10:55:04.839197 7876 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: tablet copy: Starting download of 1 WAL segments...
I20260927 10:55:04.840556 7876 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:55:04.841948 7324 tablet_copy_service.cc:342] P 4f2dee0fa9464d7ea72a69731e6112e8: Request end of tablet copy session 2604f28e264e450c8cab0f21189cd94d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.46.252@KRBTEST.COM'} at 127.0.0.1:60146
I20260927 10:55:04.842046 7324 tablet_copy_service.cc:433] P 4f2dee0fa9464d7ea72a69731e6112e8: ending tablet copy session 2604f28e264e450c8cab0f21189cd94d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2604f28e264e450c8cab0f21189cd94d
I20260927 10:55:04.849025 7876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:04.850811 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.851015 8071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.851310 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.854765 7876 server_base.cc:1034] running on GCE node
I20260927 10:55:04.854939 7876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:04.855315 7876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:04.856451 7876 hybrid_clock.cc:648] HybridClock initialized: now 1790506504856474 us; error 58 us; skew 500 ppm
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:04.857959 7876 init.cc:377] Logged in from keytab as oryx/127.1.46.252@KRBTEST.COM (short username oryx)
I20260927 10:55:04.858448 7876 webserver.cc:460] Webserver started at http://127.1.46.252:37227/ using document root <none> and password file <none>
I20260927 10:55:04.858681 7876 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.858732 7876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.860018 7876 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:04.860948 8078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.861182 7876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:04.861274 7876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
uuid: "2604f28e264e450c8cab0f21189cd94d"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "3511eec709ec41f3f6546d87210d1c32"
server_key_iv: "c9b9d4654f7ffdd98fa841329ad276b4"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.861379 7876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.866897 7876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:04.869323 7876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:04.874212 7876 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.252:35733
I20260927 10:55:04.874226 8130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.252:35733 every 8 connection(s)
I20260927 10:55:04.874887 7876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
I20260927 10:55:04.876338 8131 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:55:04.877290 8131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Bootstrap starting.
I20260927 10:55:04.884670 8131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:04.884959 8131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: Bootstrap complete.
I20260927 10:55:04.885437 8131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } }
I20260927 10:55:04.885581 8131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2604f28e264e450c8cab0f21189cd94d, State: Initialized, Role: LEARNER
I20260927 10:55:04.885699 8131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } }
I20260927 10:55:04.886008 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:04.886181 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: LEARNER
I20260927 10:55:04.886209 8131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:04.890383 8145 catalog_manager.cc:1259] Loaded cluster ID: 6fd799a375bc41b7823839243e546672
I20260927 10:55:04.890455 8145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: loading cluster ID for follower catalog manager: success
I20260927 10:55:04.891970 8145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: acquiring CA information for follower catalog manager: success
I20260927 10:55:04.893055 8145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:55:04.902689 7629 heartbeater.cc:503] Master 127.1.46.254:41443 was elected leader, sending a full tablet report...
W20260927 10:55:04.909027 8048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:04.909360 8048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:04.909422 8048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:04.916035 8048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:04.916127 8048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:04.916242 8048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:04.924607 8048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab
--principal=oryx/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:33931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=40513
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:04.925918 8048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:04.927172 8048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:04.938596 8151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.938684 8048 server_base.cc:1034] running on GCE node
W20260927 10:55:04.938596 8150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:04.938621 8153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:04.939276 8048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:04.939888 8048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:04.941039 8048 hybrid_clock.cc:648] HybridClock initialized: now 1790506504941026 us; error 26 us; skew 500 ppm
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:04.943805 8048 init.cc:377] Logged in from keytab as oryx/127.1.46.193@KRBTEST.COM (short username oryx)
I20260927 10:55:04.944692 8048 webserver.cc:460] Webserver started at http://127.1.46.193:40513/ using document root <none> and password file <none>
I20260927 10:55:04.945246 8048 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:04.945317 8048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:04.949115 8048 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:04.951648 8160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.952899 8048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:04.953009 8048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "c52462feee3e45bf8238de90cf56aaa5"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "394991086f4190df7ebf4e02496e3a79"
server_key_iv: "2abf957221dd4993aac5a265164bfc3f"
server_key_version: "encryptionkey@0"
I20260927 10:55:04.953599 8048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:04.972065 8048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:04.975194 8048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:04.975442 8048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:04.976065 8048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:04.976969 8048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:04.977020 8048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.977104 8048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:04.977149 8048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:04.987442 8048 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:33931
I20260927 10:55:04.987505 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:33931 every 8 connection(s)
I20260927 10:55:04.988507 8048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:04.997306 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 8048
I20260927 10:55:04.997520 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 7500
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.252@KRBTEST.COM
Sep 27 10:55:04 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506504, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.253@KRBTEST.COM
I20260927 10:55:05.006975 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.990533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:50285 (local address 127.1.46.252:35733)
0927 10:55:04.991044 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.991053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.992325 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 10:55:04.992998 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.993003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.993062 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.993146 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.994459 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.995213 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997043 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.997419 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997499 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.001522 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.001526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.001528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.001553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.003889 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.004896 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.004901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.004905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.005009 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.005461 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.005467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.005469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.005684 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.005817 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.006536 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.006719 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":117,"threads_started":1}
I20260927 10:55:05.007205 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:37281
--local_ip_for_outbound_sockets=127.1.46.194
--tserver_master_addrs=127.1.46.254:41443
--webserver_port=39623
--webserver_interface=127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab
--principal=oryx/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:05.008327 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.990848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:42825 (local address 127.1.46.253:38199)
0927 10:55:04.990955 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.990960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.991656 (+ 696us) server_negotiation.cc:407] Connection header received
0927 10:55:04.992870 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.992873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.992929 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.993002 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.994466 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.995287 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.996656 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.996884 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997013 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.003833 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.003840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.003846 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.003878 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.005801 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.006414 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.006421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.006423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.006520 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.006900 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.006904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.006905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.007128 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.007208 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.008035 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.008154 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 10:55:05.008733 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.991930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.1.46.193:50285)
0927 10:55:04.992214 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.992227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.992653 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.993275 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.993278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.993746 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.994296 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.994316 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.995361 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.995365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.996888 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.996899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.997660 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.998037 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.998749 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.998766 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.001328 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.004406 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.004418 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.004436 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.004718 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.005148 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.005150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.005152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.005343 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.005824 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.005833 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.006178 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.007148 (+ 970us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.007563 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":486,"spinlock_wait_cycles":120448,"thread_start_us":95,"threads_started":1}
I20260927 10:55:05.008942 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.990374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:34081 (local address 127.1.46.254:41443)
0927 10:55:04.990545 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:04.990549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:04.991657 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 10:55:04.992879 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:04.992883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:04.992952 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:04.993055 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:04.994674 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.995432 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997043 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.997400 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997502 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.002050 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.002057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.002063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.002102 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.004258 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.006399 (+ 2141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.006405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.006410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.006498 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.006858 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.006861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.006863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.007102 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.007174 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.008625 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.008738 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:55:05.009650 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.990942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.193:34081)
0927 10:55:04.991495 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.991555 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.992640 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.993248 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.993257 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.993759 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.994292 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.994311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.995549 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.995552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.996889 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.996899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.997627 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.997924 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:04.998679 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:04.998723 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.001810 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.004824 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.004832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.004837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.006292 (+ 1455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.006644 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.006649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.006654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.006746 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.007332 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.007336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.007467 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.007560 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.007717 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":306,"spinlock_wait_cycles":18688,"thread_start_us":154,"threads_started":1}
I20260927 10:55:05.009687 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:04.991024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.1.46.193:42825)
0927 10:55:04.991495 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:04.991555 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:04.992640 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:04.993248 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:04.993257 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:04.993741 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:04.994292 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.994311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.995429 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.995432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.996538 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:04.996546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:04.997043 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:04.997047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:04.999634 (+ 2587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.002032 (+ 2398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:05.002051 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.003662 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.005906 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.005909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.005911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.006263 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.006690 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.006694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.006696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.006786 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.007731 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.007734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.007825 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.008702 (+ 877us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.009243 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":371,"spinlock_wait_cycles":67712,"thread_start_us":169,"threads_started":1}
I20260927 10:55:05.010043 8274 heartbeater.cc:347] Connected to a master server at 127.1.46.252:35733
I20260927 10:55:05.010289 8275 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41443
I20260927 10:55:05.010396 8275 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.010401 8274 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.011009 8275 heartbeater.cc:511] Master 127.1.46.254:41443 requested a full tablet report, sending...
I20260927 10:55:05.011010 8274 heartbeater.cc:511] Master 127.1.46.252:35733 requested a full tablet report, sending...
I20260927 10:55:05.011543 8276 heartbeater.cc:347] Connected to a master server at 127.1.46.253:38199
I20260927 10:55:05.011627 8276 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.011799 8276 heartbeater.cc:511] Master 127.1.46.253:38199 requested a full tablet report, sending...
I20260927 10:55:05.012708 7904 ts_manager.cc:194] Registered new tserver with Master: c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931)
I20260927 10:55:05.012708 8095 ts_manager.cc:194] Registered new tserver with Master: c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931)
I20260927 10:55:05.013752 7304 ts_manager.cc:194] Re-registered known tserver with Master: c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931)
I20260927 10:55:05.014767 7304 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:34081
I20260927 10:55:05.105348 7763 heartbeater.cc:503] Master 127.1.46.254:41443 was elected leader, sending a full tablet report...
W20260927 10:55:05.119858 8282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:05.120219 8282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:05.120277 8282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:05.124250 8282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:05.124302 8282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:05.124360 8282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:05.129479 8282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab
--principal=oryx/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:37281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=39623
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:05.130357 8282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:05.131281 8282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:05.140825 8288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:05.140857 8290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:05.140829 8287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:05.141165 8282 server_base.cc:1034] running on GCE node
I20260927 10:55:05.141595 8282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:05.142227 8282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:05.143424 8282 hybrid_clock.cc:648] HybridClock initialized: now 1790506505143391 us; error 56 us; skew 500 ppm
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:05.146148 8282 init.cc:377] Logged in from keytab as oryx/127.1.46.194@KRBTEST.COM (short username oryx)
I20260927 10:55:05.147104 8282 webserver.cc:460] Webserver started at http://127.1.46.194:39623/ using document root <none> and password file <none>
I20260927 10:55:05.147692 8282 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:05.147806 8282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:05.151857 8282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:05.154094 8297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:05.155411 8282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:05.155548 8282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "916eb4920b88444aa424619c328e0fb4"
format_stamp: "Formatted at 2026-09-27 10:55:03 on dist-test-slave-cffh"
server_key: "733f8fd4ab35aa30b20cba9df1ef913a"
server_key_iv: "3597eb85dbd2705ae0e660ede2ba271e"
server_key_version: "encryptionkey@0"
I20260927 10:55:05.156129 8282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:05.170926 8282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:05.173936 8282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:05.174139 8282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:05.174821 8282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:05.175681 8282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:05.175752 8282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:05.175825 8282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:05.175868 8282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:05.185333 8282 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:37281
I20260927 10:55:05.185371 8410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:37281 every 8 connection(s)
I20260927 10:55:05.186486 8282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:05.194785 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 8282
I20260927 10:55:05.195000 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 7634
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.252@KRBTEST.COM
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.253@KRBTEST.COM
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:05.201823 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:39857
--local_ip_for_outbound_sockets=127.1.46.195
--tserver_master_addrs=127.1.46.254:41443
--webserver_port=36047
--webserver_interface=127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab
--principal=oryx/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:05.205029 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.188462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:37949 (local address 127.1.46.254:41443)
0927 10:55:05.188599 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.188602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.189834 (+ 1232us) server_negotiation.cc:407] Connection header received
0927 10:55:05.191126 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.191130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.191190 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.191286 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.192662 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.193476 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.196433 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.196877 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.196974 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.200546 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.200552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.200558 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.200591 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.202443 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.203145 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.203153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.203156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.203244 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.203679 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.203683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.203685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.203926 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.204028 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.204714 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.204838 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:55:05.206100 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.190996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:45131 (local address 127.1.46.253:38199)
0927 10:55:05.191134 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.191138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.191252 (+ 114us) server_negotiation.cc:407] Connection header received
0927 10:55:05.191365 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.191368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.191418 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.191497 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.192706 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.193478 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.196044 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.196372 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.196443 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.200202 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.200207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.200212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.200247 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.202883 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.203686 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.203692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.203695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.203784 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.204107 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.204112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.204115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.204435 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.204514 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.205794 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.205929 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:55:05.206501 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.190290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:56021 (local address 127.1.46.252:35733)
0927 10:55:05.190416 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.190421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.190546 (+ 125us) server_negotiation.cc:407] Connection header received
0927 10:55:05.190982 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.190986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.191047 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.191127 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.192700 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.193307 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.195724 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.196101 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.196168 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.201162 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.201168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.201172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.201202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.204085 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.204661 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.204664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.204666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.204753 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.205049 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.205055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.205068 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.205605 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.205710 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.206009 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.206344 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:55:05.206588 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.188841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:37949)
0927 10:55:05.189691 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.189731 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.190760 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.191433 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.191441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.191968 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:05.192522 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.192536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.193620 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.193625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.196293 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.196307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.197162 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.197168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.197294 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.197841 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:05.197860 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.200378 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.202592 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.202602 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.202620 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.203034 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.203360 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.203367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.203370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.203561 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.204092 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.204098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.204439 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.205178 (+ 739us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.205540 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":146,"spinlock_wait_cycles":17280,"thread_start_us":537,"threads_started":1}
I20260927 10:55:05.206616 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.190178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.1.46.194:56021)
0927 10:55:05.190450 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.190465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.190767 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.191431 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.191437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.191980 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:05.192550 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.192566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.193446 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.193448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.195585 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.195599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.196354 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.196358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.196558 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.197301 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:05.197330 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.200995 (+ 3665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.204220 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.204222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.204224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.204501 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.204854 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.204857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.204859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.204941 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.205707 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.205710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.205790 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.205886 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.206047 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":331,"spinlock_wait_cycles":31488,"thread_start_us":113,"threads_started":1}
I20260927 10:55:05.206676 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.190617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.1.46.194:45131)
0927 10:55:05.191165 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.191177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.191276 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.191521 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.191523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.191978 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:05.192550 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.192559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.193586 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.193589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.195918 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.195926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.196586 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.196589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.196670 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.197390 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:05.197407 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.199930 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.203272 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.203277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.203280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.203550 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.203903 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.203907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.203912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.203999 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.204658 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.204661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.204753 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.206303 (+ 1550us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.206441 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":432,"spinlock_wait_cycles":17408,"thread_start_us":114,"threads_started":1}
I20260927 10:55:05.207788 8412 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41443
I20260927 10:55:05.207880 8413 heartbeater.cc:347] Connected to a master server at 127.1.46.253:38199
I20260927 10:55:05.208019 8412 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.208676 8412 heartbeater.cc:511] Master 127.1.46.254:41443 requested a full tablet report, sending...
I20260927 10:55:05.208019 8413 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.209460 8413 heartbeater.cc:511] Master 127.1.46.253:38199 requested a full tablet report, sending...
I20260927 10:55:05.209800 7304 ts_manager.cc:194] Re-registered known tserver with Master: 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281)
I20260927 10:55:05.209815 8411 heartbeater.cc:347] Connected to a master server at 127.1.46.252:35733
I20260927 10:55:05.209949 8411 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.210124 7904 ts_manager.cc:194] Registered new tserver with Master: 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281)
I20260927 10:55:05.210237 8411 heartbeater.cc:511] Master 127.1.46.252:35733 requested a full tablet report, sending...
I20260927 10:55:05.210516 7304 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:37949
I20260927 10:55:05.211071 8095 ts_manager.cc:194] Registered new tserver with Master: 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281)
W20260927 10:55:05.315840 8418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:05.316118 8418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:05.316188 8418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:05.320257 8418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:05.320336 8418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:05.320420 8418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:05.325500 8418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab
--principal=oryx/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:39857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=36047
--tserver_master_addrs=127.1.46.254:41443,127.1.46.253:38199,127.1.46.252:35733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:05.326474 8418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:05.327482 8418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:05.336570 8423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:05.336992 8418 server_base.cc:1034] running on GCE node
W20260927 10:55:05.336589 8426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:05.336580 8424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:05.337683 8418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:05.338378 8418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:05.339875 8418 hybrid_clock.cc:648] HybridClock initialized: now 1790506505339829 us; error 51 us; skew 500 ppm
I20260927 10:55:05.342060 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.336678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.0.0.1:58750)
0927 10:55:05.337004 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.337019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.337191 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.337599 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.337602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.337620 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:05.337710 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.337716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.338731 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.338735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.339864 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.339879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.341548 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.341552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.341675 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.341705 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.341809 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.341899 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":72,"threads_started":1}
I20260927 10:55:05.342172 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.336786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.1.46.252:35733)
0927 10:55:05.336915 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.336919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.337087 (+ 168us) server_negotiation.cc:407] Connection header received
0927 10:55:05.337366 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.337370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.337454 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.337567 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:05.337846 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.338627 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.340084 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.341155 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.341273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.341474 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.341833 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.341984 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:05.342998 8105 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 10:55:05.343326 8418 init.cc:377] Logged in from keytab as oryx/127.1.46.195@KRBTEST.COM (short username oryx)
I20260927 10:55:05.343426 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f2dee0fa9464d7ea72a69731e6112e8. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:05.343603 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: LEARNER
I20260927 10:55:05.344259 8418 webserver.cc:460] Webserver started at http://127.1.46.195:36047/ using document root <none> and password file <none>
I20260927 10:55:05.344748 8418 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:05.344813 8418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:05.348342 8418 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:05.350241 8434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:05.351496 8418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:05.351601 8418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "53a1dac39e57495fad5fc4a00e2e1eed"
format_stamp: "Formatted at 2026-09-27 10:55:04 on dist-test-slave-cffh"
server_key: "4573b4e6dedba5c657506d967fc0bc6e"
server_key_iv: "db13b92b84c8a3b82e20f527bd906819"
server_key_version: "encryptionkey@0"
I20260927 10:55:05.352077 8418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:05.364539 8418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:05.367729 8418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:05.367872 8418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:05.368346 8418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:05.369107 8418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:05.369151 8418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:05.369189 8418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:05.369204 8418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:05.378739 8418 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39857
I20260927 10:55:05.378844 8547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39857 every 8 connection(s)
I20260927 10:55:05.379719 8418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:05.380517 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 8418
I20260927 10:55:05.393369 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.382572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.1.46.254:41443)
0927 10:55:05.382708 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.382715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.383679 (+ 964us) server_negotiation.cc:407] Connection header received
0927 10:55:05.383864 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.383867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.383915 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.383992 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.385002 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.385784 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.387121 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.387456 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.387521 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.389221 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.389227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.389233 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.389265 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.391202 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.391905 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.391917 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.391918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.391992 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.392269 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.392272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.392274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.392465 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.392560 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.393127 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.393231 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.252@KRBTEST.COM
I20260927 10:55:05.395875 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.383510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.1.46.252:35733)
0927 10:55:05.383638 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.383643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.384506 (+ 863us) server_negotiation.cc:407] Connection header received
0927 10:55:05.384680 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.384683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.384732 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.384802 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.385738 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.386503 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.387789 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.388447 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.388546 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.390459 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.390462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.390464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.390488 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.392196 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.393851 (+ 1655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.393854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.393856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.393949 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.394256 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.394259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.394261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.394464 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.394581 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.395590 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.395700 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.253@KRBTEST.COM
I20260927 10:55:05.397511 7304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:05.401790 7304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:05.401949 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.385285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:37481 (local address 127.1.46.252:35733)
0927 10:55:05.385810 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.385822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.385833 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:55:05.385861 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.385863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.385903 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.385983 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.387672 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.388487 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.390436 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.390865 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.390964 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.397329 (+ 6365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.397334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.397336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.397362 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.399460 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.400084 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.400088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.400089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.400179 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.400604 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.400608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.400610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.400854 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.400939 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.401633 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.401763 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":74,"threads_started":1}
Sep 27 10:55:05 dist-test-slave-cffh krb5kdc[7259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506505, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:05.403592 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.382145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.1.46.195:37481)
0927 10:55:05.383759 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.383813 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.385420 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.386291 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.386300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.386839 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:05.387514 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.387534 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.388639 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.388645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.390119 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.390132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.391341 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.391349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.392427 (+ 1078us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.393123 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:05.393141 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.397143 (+ 4002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.399591 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.399597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.399611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.399933 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.400290 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.400295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.400298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.400501 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.401007 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.401015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.401343 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.402140 (+ 797us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.402479 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"spinlock_wait_cycles":768,"thread_start_us":666,"threads_started":1}
I20260927 10:55:05.404800 8548 heartbeater.cc:347] Connected to a master server at 127.1.46.252:35733
I20260927 10:55:05.405076 8548 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.405654 8548 heartbeater.cc:511] Master 127.1.46.252:35733 requested a full tablet report, sending...
I20260927 10:55:05.406869 8095 ts_manager.cc:194] Registered new tserver with Master: 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:05.407423 8566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: NON_VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: true } } }
I20260927 10:55:05.407531 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.384118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.1.46.253:38199)
0927 10:55:05.390369 (+ 6251us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.390376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.391377 (+ 1001us) server_negotiation.cc:407] Connection header received
0927 10:55:05.391438 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.391441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.391493 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.391579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.392437 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.393209 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.395345 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.395694 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.395799 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.403178 (+ 7379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.403181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.403184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.403207 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.405188 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.405652 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.405655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.405657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.405739 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.406096 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.406099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.406100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.406320 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.407189 (+ 869us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.407247 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.407378 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6170,"thread_start_us":293,"threads_started":1}
I20260927 10:55:05.408033 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.382345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:35319 (local address 127.1.46.253:38199)
0927 10:55:05.382501 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.382508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.394535 (+ 12027us) server_negotiation.cc:407] Connection header received
0927 10:55:05.394744 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.394747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.394788 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.394835 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.395419 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.396167 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.397256 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.397534 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.397632 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.400463 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.400467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.400469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.400492 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.402310 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.403063 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.403067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.403068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.403147 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.405879 (+ 2732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.405881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.405883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.406114 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.406205 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.407736 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.407874 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 10:55:05.409057 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.382146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.1.46.195:35319)
0927 10:55:05.394453 (+ 12307us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.394468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.394627 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.394971 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.394974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.395169 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:05.395285 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.395292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.396320 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.396323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.397127 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.397134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.397734 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.397738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.397836 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.398805 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:05.398822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.400325 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.402431 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.402436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.402441 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.402930 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.403634 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.403636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.403638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.405603 (+ 1965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.406473 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.406476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.406607 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.408142 (+ 1535us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.408272 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12217,"mutex_wait_us":352,"thread_start_us":133,"threads_started":1}
I20260927 10:55:05.409104 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.394818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.195:50153)
0927 10:55:05.396561 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.396574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.397377 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.397758 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.397763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.397977 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:05.398122 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.398133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.399067 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.399070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.399872 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.399882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.400563 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.400567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.400655 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.401062 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:05.401075 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:05.402736 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:05.405413 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.405416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.405418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.405932 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.406774 (+ 842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:05.406777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:05.406779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:05.406871 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:05.408519 (+ 1648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:05.408522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:05.408615 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.408745 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.408870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1637,"thread_start_us":107,"threads_started":1}
I20260927 10:55:05.409052 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.395140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:50153 (local address 127.1.46.254:41443)
0927 10:55:05.395236 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.395239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.397313 (+ 2074us) server_negotiation.cc:407] Connection header received
0927 10:55:05.397499 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.397502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.397551 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.397606 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.398221 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.398940 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.400031 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.400333 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.400402 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.403285 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.403289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.403293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.403361 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.405198 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.406263 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.406266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.406267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.406347 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.408017 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.408021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.408022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.408210 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.408283 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.408796 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.408894 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:55:05.409921 8550 heartbeater.cc:347] Connected to a master server at 127.1.46.253:38199
I20260927 10:55:05.409996 8550 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.410156 8550 heartbeater.cc:511] Master 127.1.46.253:38199 requested a full tablet report, sending...
I20260927 10:55:05.410353 8549 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41443
I20260927 10:55:05.410421 8549 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:05.409796 8566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:05.410562 8549 heartbeater.cc:511] Master 127.1.46.254:41443 requested a full tablet report, sending...
I20260927 10:55:05.410796 7904 ts_manager.cc:194] Registered new tserver with Master: 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:05.410828 8565 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: attempting to promote NON_VOTER 2604f28e264e450c8cab0f21189cd94d to VOTER
I20260927 10:55:05.411180 8565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } }
I20260927 10:55:05.411849 7303 ts_manager.cc:194] Re-registered known tserver with Master: 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:05.412245 7914 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Refusing update from remote peer 4f2dee0fa9464d7ea72a69731e6112e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:55:05.412467 8105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 LEARNER]: Refusing update from remote peer 4f2dee0fa9464d7ea72a69731e6112e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:55:05.412592 7303 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:50153
I20260927 10:55:05.412755 8566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 10:55:05.412900 8055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 10:55:05.414939 7914 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2604f28e264e450c8cab0f21189cd94d (127.1.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.415189 8105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2604f28e264e450c8cab0f21189cd94d (127.1.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.416391 8574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.416522 8574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:05.414559 8566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2604f28e264e450c8cab0f21189cd94d (127.1.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.416716 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.416813 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:05.418161 8055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.418260 8055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:05.418202 8565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.418316 8565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:05.418504 8566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:05.418640 8566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:05.428534 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.423299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33931 (local address 127.0.0.1:44288)
0927 10:55:05.423489 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.423505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.423582 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.424968 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.424971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.424987 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:05.425052 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.425057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.426417 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.426420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.427367 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.427378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.428177 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.428181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.428261 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.428281 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.428338 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.428411 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":34}
I20260927 10:55:05.428733 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.423448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.1.46.193:33931)
0927 10:55:05.424558 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.424569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.424589 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:55:05.424658 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.424665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.424823 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.424992 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:05.425125 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.426280 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.427506 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.428058 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.428181 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.428316 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.428377 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.428479 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":109,"threads_started":1}
I20260927 10:55:05.429026 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.423892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.1.46.194:37281)
0927 10:55:05.424234 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.424249 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.424274 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:55:05.424352 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.424356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.424522 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.424671 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:05.424890 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.425684 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.427196 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.428057 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.428156 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.428287 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.428797 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.428891 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260927 10:55:05.429374 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.423455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:37281 (local address 127.0.0.1:38782)
0927 10:55:05.424099 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.424112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.424205 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.424652 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.424655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.424671 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:05.424778 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.424785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.426021 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.426027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.427054 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.427062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.428345 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.428349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.428482 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.428512 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.428582 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.429235 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":6,"thread_start_us":83,"threads_started":1}
I20260927 10:55:05.430076 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.423455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39857 (local address 127.0.0.1:54052)
0927 10:55:05.424172 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.424186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.424310 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.424855 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.424858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.424880 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:05.424992 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.424999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.426411 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.426415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.428505 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.428513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.429710 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.429713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.429796 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.429816 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.429884 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.429961 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":471,"thread_start_us":84,"threads_started":1}
I20260927 10:55:05.430498 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.423647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.1.46.195:39857)
0927 10:55:05.423918 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.423927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.424284 (+ 357us) server_negotiation.cc:407] Connection header received
0927 10:55:05.424412 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.424419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.424604 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.424738 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:05.425108 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.426279 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.428634 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.429405 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.429511 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.429648 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.430282 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.430366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
I20260927 10:55:05.431833 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 9547aa8fab1945a3a7e7282097296fc3 (DEFAULT_TABLE table=test-table [id=3cf92d69ecbb4737bce3124abbbd0a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:05.432739 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9547aa8fab1945a3a7e7282097296fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:05.432531 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 9547aa8fab1945a3a7e7282097296fc3 (DEFAULT_TABLE table=test-table [id=3cf92d69ecbb4737bce3124abbbd0a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:05.432976 8482 tablet_service.cc:1467] Processing CreateTablet for tablet 9547aa8fab1945a3a7e7282097296fc3 (DEFAULT_TABLE table=test-table [id=3cf92d69ecbb4737bce3124abbbd0a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:05.433516 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9547aa8fab1945a3a7e7282097296fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:05.433871 8482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9547aa8fab1945a3a7e7282097296fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:05.439640 8581 tablet_bootstrap.cc:492] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Bootstrap starting.
I20260927 10:55:05.441135 8582 tablet_bootstrap.cc:492] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Bootstrap starting.
I20260927 10:55:05.441830 8581 tablet_bootstrap.cc:654] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:05.442870 8581 log.cc:824] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:05.442939 8583 tablet_bootstrap.cc:492] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Bootstrap starting.
I20260927 10:55:05.443289 8582 tablet_bootstrap.cc:654] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:05.444058 8582 log.cc:824] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:05.444864 8583 tablet_bootstrap.cc:654] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:05.445182 8581 tablet_bootstrap.cc:492] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: No bootstrap required, opened a new log
I20260927 10:55:05.445359 8581 ts_tablet_manager.cc:1397] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:05.445648 8583 log.cc:824] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:05.445777 8582 tablet_bootstrap.cc:492] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: No bootstrap required, opened a new log
I20260927 10:55:05.445997 8582 ts_tablet_manager.cc:1397] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:55:05.447515 8583 tablet_bootstrap.cc:492] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: No bootstrap required, opened a new log
I20260927 10:55:05.447697 8583 ts_tablet_manager.cc:1397] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:05.450080 8582 raft_consensus.cc:348] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.450299 8582 raft_consensus.cc:374] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:05.450354 8582 raft_consensus.cc:729] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916eb4920b88444aa424619c328e0fb4, State: Initialized, Role: FOLLOWER
I20260927 10:55:05.450384 8581 raft_consensus.cc:348] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.450640 8581 raft_consensus.cc:374] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:05.450701 8581 raft_consensus.cc:729] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52462feee3e45bf8238de90cf56aaa5, State: Initialized, Role: FOLLOWER
I20260927 10:55:05.451067 8582 consensus_queue.cc:260] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.451116 8581 consensus_queue.cc:260] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.451980 8582 ts_tablet_manager.cc:1428] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:55:05.451980 8581 ts_tablet_manager.cc:1428] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260927 10:55:05.452106 8583 raft_consensus.cc:348] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.452317 8583 raft_consensus.cc:374] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:05.452369 8583 raft_consensus.cc:729] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a1dac39e57495fad5fc4a00e2e1eed, State: Initialized, Role: FOLLOWER
I20260927 10:55:05.452656 8275 heartbeater.cc:503] Master 127.1.46.254:41443 was elected leader, sending a full tablet report...
I20260927 10:55:05.452754 8583 consensus_queue.cc:260] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.453361 8549 heartbeater.cc:503] Master 127.1.46.254:41443 was elected leader, sending a full tablet report...
I20260927 10:55:05.455468 8583 ts_tablet_manager.cc:1428] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 10:55:05.457613 8412 heartbeater.cc:503] Master 127.1.46.254:41443 was elected leader, sending a full tablet report...
W20260927 10:55:05.490535 8277 tablet.cc:2369] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:55:05.631001 8551 tablet.cc:2369] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:05.654884 8588 raft_consensus.cc:482] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:05.655133 8588 raft_consensus.cc:504] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.656423 8588 leader_election.cc:288] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281), 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:05.660540 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.656783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:39073 (local address 127.1.46.195:39857)
0927 10:55:05.656910 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.656915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.657177 (+ 262us) server_negotiation.cc:407] Connection header received
0927 10:55:05.657323 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.657327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.657382 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.657468 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:05.657725 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.658369 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.659505 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.659971 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.660096 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.660150 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.660352 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.660420 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:55:05.660625 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.656818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:43701 (local address 127.1.46.194:37281)
0927 10:55:05.656939 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.656943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.657158 (+ 215us) server_negotiation.cc:407] Connection header received
0927 10:55:05.657330 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.657333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.657373 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.657452 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:05.657750 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.658319 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.659462 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.660071 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.660192 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.660241 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.660438 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.660499 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:55:05.660670 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.656725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:37281 (local address 127.1.46.193:43701)
0927 10:55:05.657061 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.657076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.657206 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.657462 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.657465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.657488 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:05.657647 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.657653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.658435 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.658439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.659343 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.659351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.660181 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.660185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.660310 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.660340 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.660424 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.660499 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":43,"thread_start_us":68,"threads_started":1}
I20260927 10:55:05.660557 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.656743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39857 (local address 127.1.46.193:39073)
0927 10:55:05.657087 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:05.657098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:05.657217 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:05.657478 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:05.657481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:05.657495 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:05.657623 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.657629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.658477 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.658484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.659334 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:05.659342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.660084 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.660088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:05.660181 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.660239 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 10:55:05.660323 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:55:05.660393 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":28,"thread_start_us":71,"threads_started":1}
I20260927 10:55:05.661247 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9547aa8fab1945a3a7e7282097296fc3" candidate_uuid: "c52462feee3e45bf8238de90cf56aaa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916eb4920b88444aa424619c328e0fb4" is_pre_election: true
I20260927 10:55:05.661449 8502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9547aa8fab1945a3a7e7282097296fc3" candidate_uuid: "c52462feee3e45bf8238de90cf56aaa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" is_pre_election: true
I20260927 10:55:05.661566 8365 raft_consensus.cc:2454] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c52462feee3e45bf8238de90cf56aaa5 in term 0.
I20260927 10:55:05.661691 8502 raft_consensus.cc:2454] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c52462feee3e45bf8238de90cf56aaa5 in term 0.
I20260927 10:55:05.661998 8162 leader_election.cc:302] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916eb4920b88444aa424619c328e0fb4, c52462feee3e45bf8238de90cf56aaa5; no voters:
I20260927 10:55:05.662271 8588 raft_consensus.cc:2781] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:05.662385 8588 raft_consensus.cc:482] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:05.662446 8588 raft_consensus.cc:3037] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:05.663475 8588 raft_consensus.cc:504] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.663801 8588 leader_election.cc:288] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [CANDIDATE]: Term 1 election: Requested vote from peers 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281), 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:05.664160 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9547aa8fab1945a3a7e7282097296fc3" candidate_uuid: "c52462feee3e45bf8238de90cf56aaa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916eb4920b88444aa424619c328e0fb4"
I20260927 10:55:05.664181 8502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9547aa8fab1945a3a7e7282097296fc3" candidate_uuid: "c52462feee3e45bf8238de90cf56aaa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a1dac39e57495fad5fc4a00e2e1eed"
I20260927 10:55:05.664278 8365 raft_consensus.cc:3037] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:05.664324 8502 raft_consensus.cc:3037] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:05.665261 8365 raft_consensus.cc:2454] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c52462feee3e45bf8238de90cf56aaa5 in term 1.
I20260927 10:55:05.665338 8502 raft_consensus.cc:2454] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c52462feee3e45bf8238de90cf56aaa5 in term 1.
I20260927 10:55:05.665560 8162 leader_election.cc:302] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916eb4920b88444aa424619c328e0fb4, c52462feee3e45bf8238de90cf56aaa5; no voters:
I20260927 10:55:05.665715 8588 raft_consensus.cc:2781] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:05.665863 8588 raft_consensus.cc:686] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 1 LEADER]: Becoming Leader. State: Replica: c52462feee3e45bf8238de90cf56aaa5, State: Running, Role: LEADER
I20260927 10:55:05.666182 8588 consensus_queue.cc:237] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:05.669503 7304 catalog_manager.cc:5495] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193). New cstate: current_term: 1 leader_uuid: "c52462feee3e45bf8238de90cf56aaa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } health_report { overall_health: UNKNOWN } } }
W20260927 10:55:05.688184 8414 tablet.cc:2369] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:05.727320 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.723787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.1.46.193:33931)
0927 10:55:05.723905 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.723909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.723922 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:05.724045 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.724047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.724103 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.724208 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:05.724406 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.724904 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.725765 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.725992 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.726074 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.726430 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:05.726953 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:05.727061 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.727164 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.727214 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:55:05.733443 8502 raft_consensus.cc:1261] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Refusing update from remote peer c52462feee3e45bf8238de90cf56aaa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:05.733443 8365 raft_consensus.cc:1261] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 1 FOLLOWER]: Refusing update from remote peer c52462feee3e45bf8238de90cf56aaa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:05.734304 8592 consensus_queue.cc:1035] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:05.734524 8588 consensus_queue.cc:1035] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:05.743119 8597 mvcc.cc:205] Tried to move back new op lower bound from 7333914647475675136 to 7333914647211544576. Current Snapshot: MvccSnapshot[applied={T|T < 7333914647475675136}]
I20260927 10:55:05.745309 8595 mvcc.cc:205] Tried to move back new op lower bound from 7333914647475675136 to 7333914647211544576. Current Snapshot: MvccSnapshot[applied={T|T < 7333914647475675136}]
I20260927 10:55:05.746688 8596 mvcc.cc:205] Tried to move back new op lower bound from 7333914647475675136 to 7333914647211544576. Current Snapshot: MvccSnapshot[applied={T|T < 7333914647475675136}]
I20260927 10:55:05.753325 7302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:05.753501 7302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:05.757839 7302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f2dee0fa9464d7ea72a69731e6112e8: Sending DeleteTablet for 3 replicas of tablet 9547aa8fab1945a3a7e7282097296fc3
I20260927 10:55:05.758510 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 9547aa8fab1945a3a7e7282097296fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:05 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:38782
I20260927 10:55:05.758791 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 9547aa8fab1945a3a7e7282097296fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:05 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44288
I20260927 10:55:05.759297 8605 tablet_replica.cc:330] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: stopping tablet replica
I20260927 10:55:05.759670 8605 raft_consensus.cc:2229] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:05.760018 8605 raft_consensus.cc:2258] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:05.761650 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 7272
I20260927 10:55:05.763512 8605 ts_tablet_manager.cc:1905] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:05.766561 8605 ts_tablet_manager.cc:1918] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:05.766685 8605 log.cc:1196] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/9547aa8fab1945a3a7e7282097296fc3
I20260927 10:55:05.766943 8604 tablet_replica.cc:330] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: stopping tablet replica
I20260927 10:55:05.767037 8605 ts_tablet_manager.cc:1939] T 9547aa8fab1945a3a7e7282097296fc3 P c52462feee3e45bf8238de90cf56aaa5: Deleting consensus metadata
I20260927 10:55:05.767426 8604 raft_consensus.cc:2229] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:05.767858 8604 raft_consensus.cc:2258] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:55:05.768476 8163 connection.cc:535] server connection from 127.0.0.1:44288 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 10:55:05.768850 7888 connection.cc:535] server connection from 127.0.0.1:40290 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:55:05.770674 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.759820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.1.46.253:38199)
0927 10:55:05.759938 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.759943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.760374 (+ 431us) server_negotiation.cc:407] Connection header received
0927 10:55:05.760608 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.760611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.760659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.760710 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.761738 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.762444 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.763788 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.764105 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.764203 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.766840 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.766846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.766853 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.766884 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.768697 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.769184 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.769188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.769190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.769271 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.769622 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.769626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.769627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.769842 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.769949 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.770299 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.770445 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:55:05.771021 8604 ts_tablet_manager.cc:1905] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:05.773850 8604 ts_tablet_manager.cc:1918] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:05.773952 8604 log.cc:1196] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/9547aa8fab1945a3a7e7282097296fc3
I20260927 10:55:05.774293 8604 ts_tablet_manager.cc:1939] T 9547aa8fab1945a3a7e7282097296fc3 P 916eb4920b88444aa424619c328e0fb4: Deleting consensus metadata
W20260927 10:55:05.775410 8300 connection.cc:441] server connection from 127.0.0.1:38782 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38782 (request call id 1) could send its response
I20260927 10:55:05.782393 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.772673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.1.46.252:35733)
0927 10:55:05.772775 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.772779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.773278 (+ 499us) server_negotiation.cc:407] Connection header received
0927 10:55:05.773427 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.773430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.773467 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.773514 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.774657 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.775390 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.776568 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.776862 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.776954 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.778613 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.778616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.778619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.778643 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.780500 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.780902 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.780908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.780913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.781005 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.781369 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.781374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.781376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.781615 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.781756 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.782059 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.782187 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:55:05.782971 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.760253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.1.46.252:35733)
0927 10:55:05.760356 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.760360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.772102 (+ 11742us) server_negotiation.cc:407] Connection header received
0927 10:55:05.772296 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.772299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.772357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.772429 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.773340 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.774167 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.775507 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.775838 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.775927 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.778002 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.778006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.778009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.778033 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.780992 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.781691 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.781694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.781696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.781776 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.782085 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.782089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.782090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.782277 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.782369 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.782716 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.782828 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 10:55:05.786041 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:05.776671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.1.46.253:38199)
0927 10:55:05.776765 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:05.776769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:05.777074 (+ 305us) server_negotiation.cc:407] Connection header received
0927 10:55:05.777226 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:05.777229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:05.777279 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:05.777334 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:05.778251 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.779065 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.780399 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:05.780734 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:05.780832 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:05.782793 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:05.782796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:05.782799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:05.782820 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:05.784594 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.784986 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.784992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.784996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.785083 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:05.785353 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:05.785356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:05.785357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:05.785505 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:05.785594 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:05.785813 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 10:55:05.785891 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:55:06.746239 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:06.745799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.195:54549)
0927 10:55:06.746081 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:06.746153 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":71,"threads_started":1}
W20260927 10:55:06.746686 8549 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:41443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:06.749622 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:06.749051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.193:44991)
0927 10:55:06.749424 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:06.749490 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":71,"threads_started":1}
W20260927 10:55:06.749934 8275 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:41443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:06.750119 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:06.749589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:53059)
0927 10:55:06.749872 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:06.749995 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":67,"threads_started":1}
W20260927 10:55:06.750483 8412 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:41443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:07.243505 8616 raft_consensus.cc:482] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c52462feee3e45bf8238de90cf56aaa5)
I20260927 10:55:07.243647 8616 raft_consensus.cc:504] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.244803 8616 leader_election.cc:288] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281), c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931)
I20260927 10:55:07.248500 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.245211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:58169 (local address 127.1.46.193:33931)
0927 10:55:07.245425 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.245428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.245449 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:55:07.245492 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.245494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.245569 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.245668 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.245953 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.246434 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.247501 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.247882 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.248000 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.248068 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.248316 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.248387 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
I20260927 10:55:07.248492 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.244952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:37281 (local address 127.1.46.195:52757)
0927 10:55:07.245092 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.245108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.245204 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.245605 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.245611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.245651 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.245819 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.245825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.246628 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.246632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.247357 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.247365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.247997 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.248000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.248143 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.248191 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.248248 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.248332 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":83}
I20260927 10:55:07.248494 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.245084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:52757 (local address 127.1.46.194:37281)
0927 10:55:07.245329 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.245335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.245354 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:55:07.245407 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.245410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.245463 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.245550 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.245939 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.246444 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.247476 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.247881 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.248017 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.248072 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.248290 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.248359 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260927 10:55:07.248520 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.245063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33931 (local address 127.1.46.195:58169)
0927 10:55:07.245309 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.245322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.245411 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.245706 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.245708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.245722 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.245829 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.245833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.246562 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.246588 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.247361 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.247369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.248006 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.248009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.248182 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.248200 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.248289 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.248354 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":71,"thread_start_us":57,"threads_started":1}
W20260927 10:55:07.249378 8436 leader_election.cc:341] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281): Not found: Tablet not found: 9547aa8fab1945a3a7e7282097296fc3
W20260927 10:55:07.249621 8437 leader_election.cc:341] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931): Not found: Tablet not found: 9547aa8fab1945a3a7e7282097296fc3
I20260927 10:55:07.249727 8437 leader_election.cc:302] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53a1dac39e57495fad5fc4a00e2e1eed; no voters: 916eb4920b88444aa424619c328e0fb4, c52462feee3e45bf8238de90cf56aaa5
I20260927 10:55:07.250051 8616 raft_consensus.cc:2726] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:55:07.258767 8620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4f2dee0fa9464d7ea72a69731e6112e8)
I20260927 10:55:07.258953 8620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } }
I20260927 10:55:07.259968 8620 leader_election.cc:288] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f2dee0fa9464d7ea72a69731e6112e8 (127.1.46.254:41443), 2604f28e264e450c8cab0f21189cd94d (127.1.46.252:35733)
I20260927 10:55:07.260505 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.260041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60604)
0927 10:55:07.260309 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.260386 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
W20260927 10:55:07.260715 7891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:07.261147 8625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4f2dee0fa9464d7ea72a69731e6112e8)
I20260927 10:55:07.261312 8625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } }
I20260927 10:55:07.261570 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.261324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60618)
0927 10:55:07.261437 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.261496 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 10:55:07.261752 7891 leader_election.cc:334] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4f2dee0fa9464d7ea72a69731e6112e8 (127.1.46.254:41443): Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:07.262558 8625 leader_election.cc:288] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f2dee0fa9464d7ea72a69731e6112e8 (127.1.46.254:41443), 3f45cab475a749b48659c4108a524459 (127.1.46.253:38199)
I20260927 10:55:07.262698 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.262270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60624)
0927 10:55:07.262458 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.262543 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":55,"threads_started":1}
W20260927 10:55:07.262876 8082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:07.263679 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.263418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60632)
0927 10:55:07.263560 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.263605 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 10:55:07.263767 8082 leader_election.cc:334] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4f2dee0fa9464d7ea72a69731e6112e8 (127.1.46.254:41443): Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:07.264835 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.260196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:35733 (local address 127.0.0.1:58808)
0927 10:55:07.260397 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.260413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.260520 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.261119 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.261124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.261142 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.261273 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.261280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.262299 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.262303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.263694 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.263700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.264397 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.264402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.264486 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.264516 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.264646 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.264715 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260927 10:55:07.264856 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.260378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.1.46.252:35733)
0927 10:55:07.260756 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.260761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.260778 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:55:07.260828 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.260831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.260936 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.261041 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.261372 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.262180 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.263824 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.264278 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.264406 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.264461 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.264671 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.264744 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260927 10:55:07.265283 8105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f45cab475a749b48659c4108a524459" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2604f28e264e450c8cab0f21189cd94d" is_pre_election: true
I20260927 10:55:07.265479 8105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f45cab475a749b48659c4108a524459 in term 1.
I20260927 10:55:07.265831 7889 leader_election.cc:302] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2604f28e264e450c8cab0f21189cd94d, 3f45cab475a749b48659c4108a524459; no voters: 4f2dee0fa9464d7ea72a69731e6112e8
I20260927 10:55:07.266033 8620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:55:07.266089 8620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4f2dee0fa9464d7ea72a69731e6112e8)
I20260927 10:55:07.266129 8620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:55:07.266726 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.262624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.1.46.253:38199)
0927 10:55:07.263282 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.263290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.263302 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:07.263338 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.263340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.263375 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.263457 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.263701 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.264254 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.265591 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.266194 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.266323 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.266373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.266550 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.266632 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":98,"threads_started":1}
I20260927 10:55:07.266727 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.262323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:38199 (local address 127.0.0.1:40330)
0927 10:55:07.262787 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.262802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.262882 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.263479 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.263482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.263502 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.263591 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.263597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.264364 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.264369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.265470 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.265485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.266322 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.266326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.266415 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.266450 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.266537 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.266617 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":73,"threads_started":1}
I20260927 10:55:07.267123 7914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2604f28e264e450c8cab0f21189cd94d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f45cab475a749b48659c4108a524459" is_pre_election: true
I20260927 10:55:07.267386 8620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } }
I20260927 10:55:07.267647 7914 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2604f28e264e450c8cab0f21189cd94d in current term 2: Already voted for candidate 3f45cab475a749b48659c4108a524459 in this term.
I20260927 10:55:07.267753 8620 leader_election.cc:288] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [CANDIDATE]: Term 2 election: Requested vote from peers 4f2dee0fa9464d7ea72a69731e6112e8 (127.1.46.254:41443), 2604f28e264e450c8cab0f21189cd94d (127.1.46.252:35733)
I20260927 10:55:07.268106 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.267834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60640)
0927 10:55:07.267966 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.268015 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 10:55:07.268194 8105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f45cab475a749b48659c4108a524459" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2604f28e264e450c8cab0f21189cd94d"
I20260927 10:55:07.268244 8079 leader_election.cc:302] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2604f28e264e450c8cab0f21189cd94d; no voters: 3f45cab475a749b48659c4108a524459, 4f2dee0fa9464d7ea72a69731e6112e8
I20260927 10:55:07.268296 8105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:55:07.268754 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.268512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60646)
0927 10:55:07.268645 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.268694 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:55:07.268869 7891 leader_election.cc:334] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4f2dee0fa9464d7ea72a69731e6112e8 (127.1.46.254:41443): Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
I20260927 10:55:07.269402 8105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f45cab475a749b48659c4108a524459 in term 2.
I20260927 10:55:07.269534 8625 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:55:07.269761 7889 leader_election.cc:302] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2604f28e264e450c8cab0f21189cd94d, 3f45cab475a749b48659c4108a524459; no voters: 4f2dee0fa9464d7ea72a69731e6112e8
I20260927 10:55:07.269935 8620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:55:07.270176 8620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [term 2 LEADER]: Becoming Leader. State: Replica: 3f45cab475a749b48659c4108a524459, State: Running, Role: LEADER
I20260927 10:55:07.270429 8620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } }
I20260927 10:55:07.271399 8630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f45cab475a749b48659c4108a524459. Latest consensus state: current_term: 2 leader_uuid: "3f45cab475a749b48659c4108a524459" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:07.271534 8630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:07.271806 8632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:07.273880 8632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:07.274260 8632 catalog_manager.cc:1259] Loaded cluster ID: 6fd799a375bc41b7823839243e546672
I20260927 10:55:07.274304 8632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:07.274881 8632 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:07.275249 8632 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Loaded TSK: 0
I20260927 10:55:07.275902 8632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:55:07.301162 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:07.304176 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:07.307044 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.306814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60654)
0927 10:55:07.306926 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.306981 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:55:07.307119 8105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [term 2 FOLLOWER]: Refusing update from remote peer 3f45cab475a749b48659c4108a524459: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 10:55:07.307583 8630 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 10:55:07.307662 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.307438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.0.0.1:60670)
0927 10:55:07.307562 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.307610 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:55:07.307811 7891 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 -> Peer 4f2dee0fa9464d7ea72a69731e6112e8 (127.1.46.254:41443): Couldn't send request to peer 4f2dee0fa9464d7ea72a69731e6112e8. Status: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:55:07.308985 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f45cab475a749b48659c4108a524459. Latest consensus state: current_term: 2 leader_uuid: "3f45cab475a749b48659c4108a524459" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:07.309124 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:07.310036 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3f45cab475a749b48659c4108a524459" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:07.310148 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:07.310747 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3f45cab475a749b48659c4108a524459" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f2dee0fa9464d7ea72a69731e6112e8" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41443 } } peers { permanent_uuid: "3f45cab475a749b48659c4108a524459" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 38199 } attrs { promote: false } } peers { permanent_uuid: "2604f28e264e450c8cab0f21189cd94d" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 35733 } attrs { promote: false } } }
I20260927 10:55:07.310863 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604f28e264e450c8cab0f21189cd94d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:55:07.311070 8634 mvcc.cc:205] Tried to move back new op lower bound from 7333914653925355520 to 7333914653781729280. Current Snapshot: MvccSnapshot[applied={T|T < 7333914653925355520 or (T in {7333914653925355520})}]
I20260927 10:55:07.319605 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.315296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33931 (local address 127.0.0.1:44312)
0927 10:55:07.315764 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.315777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.315873 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.316271 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.316275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.316291 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.316371 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.316375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.317291 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.317296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.318187 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.318195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.319161 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.319167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.319266 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.319293 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.319387 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.319457 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":21}
I20260927 10:55:07.319815 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.315409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.1.46.193:33931)
0927 10:55:07.315534 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.315539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.315856 (+ 317us) server_negotiation.cc:407] Connection header received
0927 10:55:07.315976 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.315980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.316038 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.316158 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.316475 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.317212 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.318317 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.318958 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.319051 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.319109 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.319602 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.319685 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:55:07.320418 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.315480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:37281 (local address 127.0.0.1:38790)
0927 10:55:07.316163 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.316178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.316587 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.316856 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.316859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.316875 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.316957 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.316967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.317979 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.317983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.319132 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.319142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.320064 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.320070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.320158 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.320179 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.320248 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.320312 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":6}
I20260927 10:55:07.320372 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 846bef9fa183401d8842abcd7fe9c8ea (DEFAULT_TABLE table=test-table [id=90edf74209ab4231b130feefdb56ad65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:07.320670 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846bef9fa183401d8842abcd7fe9c8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:07.320762 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.315624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.1.46.194:37281)
0927 10:55:07.316159 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.316163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.316230 (+ 67us) server_negotiation.cc:407] Connection header received
0927 10:55:07.316693 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.316697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.316749 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.316826 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.317095 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.317862 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.319243 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.319842 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.319942 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.320007 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.320563 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.320642 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260927 10:55:07.323171 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 846bef9fa183401d8842abcd7fe9c8ea (DEFAULT_TABLE table=test-table [id=90edf74209ab4231b130feefdb56ad65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:07.323446 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846bef9fa183401d8842abcd7fe9c8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:07.323603 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.315817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39857 (local address 127.0.0.1:54054)
0927 10:55:07.316525 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.316538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.316610 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.317047 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.317050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.317065 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.317136 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.317141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.318113 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.318118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.318931 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.318939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.323233 (+ 4294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.323240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.323334 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.323353 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.323418 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.323486 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":84,"threads_started":1}
I20260927 10:55:07.323652 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.316291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.1.46.195:39857)
0927 10:55:07.316814 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.316829 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.316844 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:55:07.316889 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.316891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.316936 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.317029 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.317246 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.318000 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.319062 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.323113 (+ 4051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.323321 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.323402 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.323439 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.323514 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":69,"threads_started":1}
I20260927 10:55:07.324070 8482 tablet_service.cc:1467] Processing CreateTablet for tablet 846bef9fa183401d8842abcd7fe9c8ea (DEFAULT_TABLE table=test-table [id=90edf74209ab4231b130feefdb56ad65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:07.324328 8482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846bef9fa183401d8842abcd7fe9c8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:07.325639 8642 tablet_bootstrap.cc:492] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: Bootstrap starting.
I20260927 10:55:07.325651 8641 tablet_bootstrap.cc:492] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5: Bootstrap starting.
I20260927 10:55:07.326375 8641 tablet_bootstrap.cc:654] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:07.326536 8642 tablet_bootstrap.cc:654] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:07.326784 8643 tablet_bootstrap.cc:492] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: Bootstrap starting.
I20260927 10:55:07.327630 8641 tablet_bootstrap.cc:492] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5: No bootstrap required, opened a new log
I20260927 10:55:07.327646 8643 tablet_bootstrap.cc:654] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:07.327710 8642 tablet_bootstrap.cc:492] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: No bootstrap required, opened a new log
I20260927 10:55:07.327720 8641 ts_tablet_manager.cc:1397] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 10:55:07.327786 8642 ts_tablet_manager.cc:1397] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:07.328147 8641 raft_consensus.cc:348] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.328181 8642 raft_consensus.cc:348] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.328274 8642 raft_consensus.cc:374] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:07.328274 8641 raft_consensus.cc:374] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:07.328307 8641 raft_consensus.cc:729] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52462feee3e45bf8238de90cf56aaa5, State: Initialized, Role: FOLLOWER
I20260927 10:55:07.328308 8642 raft_consensus.cc:729] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916eb4920b88444aa424619c328e0fb4, State: Initialized, Role: FOLLOWER
I20260927 10:55:07.328442 8642 consensus_queue.cc:260] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.328454 8641 consensus_queue.cc:260] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.328778 8642 ts_tablet_manager.cc:1428] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:07.328923 8643 tablet_bootstrap.cc:492] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: No bootstrap required, opened a new log
I20260927 10:55:07.329015 8643 ts_tablet_manager.cc:1397] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:07.329625 8643 raft_consensus.cc:348] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.329744 8643 raft_consensus.cc:374] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:07.329780 8643 raft_consensus.cc:729] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a1dac39e57495fad5fc4a00e2e1eed, State: Initialized, Role: FOLLOWER
I20260927 10:55:07.329900 8643 consensus_queue.cc:260] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.331601 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.331349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.195:51891)
0927 10:55:07.331470 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.331523 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:55:07.331717 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.329174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:52469)
0927 10:55:07.331465 (+ 2291us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.331539 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2204,"thread_start_us":84,"threads_started":1}
I20260927 10:55:07.331831 8643 ts_tablet_manager.cc:1428] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:55:07.331974 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.331606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.193:47945)
0927 10:55:07.331836 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.331890 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260927 10:55:07.332655 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.332240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:42391)
0927 10:55:07.332486 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.332554 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
I20260927 10:55:07.332935 8413 heartbeater.cc:503] Master 127.1.46.253:38199 was elected leader, sending a full tablet report...
I20260927 10:55:07.333375 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.333154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.195:40843)
0927 10:55:07.333263 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.333312 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 10:55:07.333416 8641 ts_tablet_manager.cc:1428] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 10:55:07.333997 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.333739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.193:35925)
0927 10:55:07.333888 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.333933 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 10:55:07.333379 8550 heartbeater.cc:503] Master 127.1.46.253:38199 was elected leader, sending a full tablet report...
I20260927 10:55:07.335604 8276 heartbeater.cc:503] Master 127.1.46.253:38199 was elected leader, sending a full tablet report...
I20260927 10:55:07.336298 8482 tablet_service.cc:1514] Processing DeleteTablet for tablet 9547aa8fab1945a3a7e7282097296fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:05 UTC) from {username='oryx', principal='oryx/127.1.46.253@KRBTEST.COM'} at 127.0.0.1:54054
I20260927 10:55:07.336738 8652 tablet_replica.cc:330] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: stopping tablet replica
I20260927 10:55:07.337066 8652 raft_consensus.cc:2229] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:07.337378 8652 raft_consensus.cc:2258] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:07.338279 8652 ts_tablet_manager.cc:1905] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:07.340209 8652 ts_tablet_manager.cc:1918] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:07.340283 8652 log.cc:1196] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/9547aa8fab1945a3a7e7282097296fc3
I20260927 10:55:07.340588 8652 ts_tablet_manager.cc:1939] T 9547aa8fab1945a3a7e7282097296fc3 P 53a1dac39e57495fad5fc4a00e2e1eed: Deleting consensus metadata
I20260927 10:55:07.341539 7888 catalog_manager.cc:4841] TS 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857): tablet 9547aa8fab1945a3a7e7282097296fc3 (table test-table [id=3cf92d69ecbb4737bce3124abbbd0a6a]) successfully deleted
I20260927 10:55:07.356835 8648 raft_consensus.cc:482] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:07.356988 8648 raft_consensus.cc:504] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.358139 8648 leader_election.cc:288] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931), 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:07.363135 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.358495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:59039 (local address 127.1.46.195:39857)
0927 10:55:07.358643 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.358647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.358664 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:55:07.358712 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.358716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.358767 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.358846 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.359263 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.360053 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.361828 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.362431 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.362555 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.362632 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.362958 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.363022 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:55:07.363147 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.358439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:44821 (local address 127.1.46.193:33931)
0927 10:55:07.358554 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.358560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.358683 (+ 123us) server_negotiation.cc:407] Connection header received
0927 10:55:07.358777 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.358781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.358836 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.358902 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:07.359263 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.360065 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.361828 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.362443 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.362552 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.362632 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.362958 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.363020 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:55:07.363186 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.358362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39857 (local address 127.1.46.194:59039)
0927 10:55:07.358453 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.358467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.358614 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.358869 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.358872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.358922 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.359153 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.359160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.360166 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.360170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.361704 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.361715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.362587 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.362591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.362761 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.362853 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.362934 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.363035 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":237}
I20260927 10:55:07.363185 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.358362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33931 (local address 127.1.46.194:44821)
0927 10:55:07.358600 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.358613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:07.358688 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:07.358931 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:07.358933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:07.358948 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:07.359153 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.359160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.360166 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.360170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.361704 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:07.361715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.362587 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.362591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:07.362782 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.362853 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 10:55:07.362934 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:55:07.363037 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":405}
I20260927 10:55:07.363610 8502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846bef9fa183401d8842abcd7fe9c8ea" candidate_uuid: "916eb4920b88444aa424619c328e0fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" is_pre_election: true
I20260927 10:55:07.363752 8502 raft_consensus.cc:2454] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916eb4920b88444aa424619c328e0fb4 in term 0.
I20260927 10:55:07.363890 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846bef9fa183401d8842abcd7fe9c8ea" candidate_uuid: "916eb4920b88444aa424619c328e0fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c52462feee3e45bf8238de90cf56aaa5" is_pre_election: true
I20260927 10:55:07.364109 8228 raft_consensus.cc:2454] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916eb4920b88444aa424619c328e0fb4 in term 0.
I20260927 10:55:07.364125 8298 leader_election.cc:302] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a1dac39e57495fad5fc4a00e2e1eed, 916eb4920b88444aa424619c328e0fb4; no voters:
I20260927 10:55:07.364377 8648 raft_consensus.cc:2781] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:07.364439 8648 raft_consensus.cc:482] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:07.364490 8648 raft_consensus.cc:3037] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:07.365377 8648 raft_consensus.cc:504] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.365715 8648 leader_election.cc:288] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [CANDIDATE]: Term 1 election: Requested vote from peers c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931), 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857)
I20260927 10:55:07.366041 8502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846bef9fa183401d8842abcd7fe9c8ea" candidate_uuid: "916eb4920b88444aa424619c328e0fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a1dac39e57495fad5fc4a00e2e1eed"
I20260927 10:55:07.366041 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846bef9fa183401d8842abcd7fe9c8ea" candidate_uuid: "916eb4920b88444aa424619c328e0fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c52462feee3e45bf8238de90cf56aaa5"
I20260927 10:55:07.366150 8502 raft_consensus.cc:3037] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:07.366150 8228 raft_consensus.cc:3037] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:07.367192 8502 raft_consensus.cc:2454] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916eb4920b88444aa424619c328e0fb4 in term 1.
I20260927 10:55:07.367329 8228 raft_consensus.cc:2454] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916eb4920b88444aa424619c328e0fb4 in term 1.
I20260927 10:55:07.367501 8298 leader_election.cc:302] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a1dac39e57495fad5fc4a00e2e1eed, 916eb4920b88444aa424619c328e0fb4; no voters:
I20260927 10:55:07.367684 8648 raft_consensus.cc:2781] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:07.367965 8648 raft_consensus.cc:686] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 1 LEADER]: Becoming Leader. State: Replica: 916eb4920b88444aa424619c328e0fb4, State: Running, Role: LEADER
I20260927 10:55:07.368238 8648 consensus_queue.cc:237] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } }
I20260927 10:55:07.368625 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.368408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:34513)
0927 10:55:07.368501 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.368570 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 10:55:07.370621 7903 catalog_manager.cc:5495] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 916eb4920b88444aa424619c328e0fb4 (127.1.46.194). New cstate: current_term: 1 leader_uuid: "916eb4920b88444aa424619c328e0fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "916eb4920b88444aa424619c328e0fb4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 } health_report { overall_health: UNKNOWN } } }
I20260927 10:55:07.388216 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.384457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.1.46.194:37281)
0927 10:55:07.384567 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:07.384571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:07.384586 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:55:07.384734 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:07.384737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:07.384793 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:07.384897 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:07.385080 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.385880 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.386879 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:07.387190 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:07.387281 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:07.387474 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:07.387902 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:07.387981 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:07.388047 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 10:55:07.388088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:55:07.390671 8502 raft_consensus.cc:1261] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Refusing update from remote peer 916eb4920b88444aa424619c328e0fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:07.390997 8228 raft_consensus.cc:1261] T 846bef9fa183401d8842abcd7fe9c8ea P c52462feee3e45bf8238de90cf56aaa5 [term 1 FOLLOWER]: Refusing update from remote peer 916eb4920b88444aa424619c328e0fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:07.391137 8648 consensus_queue.cc:1035] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53a1dac39e57495fad5fc4a00e2e1eed" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:07.391521 8653 consensus_queue.cc:1035] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c52462feee3e45bf8238de90cf56aaa5" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:07.392628 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.392342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.195:49167)
0927 10:55:07.392494 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.392561 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 10:55:07.393579 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.393354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.193:38305)
0927 10:55:07.393451 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.393506 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 10:55:07.394984 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.394762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:38767)
0927 10:55:07.394869 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.394920 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:55:07.397471 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.397205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.195:60077)
0927 10:55:07.397360 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.397409 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 10:55:07.397845 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.397607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.193:32985)
0927 10:55:07.397739 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.397783 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 10:55:07.398442 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.398194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41443 (local address 127.1.46.194:56967)
0927 10:55:07.398340 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.398384 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 10:55:07.404549 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:07.404714 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:07.408201 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f45cab475a749b48659c4108a524459: Sending DeleteTablet for 3 replicas of tablet 846bef9fa183401d8842abcd7fe9c8ea
I20260927 10:55:07.408776 8482 tablet_service.cc:1514] Processing DeleteTablet for tablet 846bef9fa183401d8842abcd7fe9c8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:07 UTC) from {username='oryx', principal='oryx/127.1.46.253@KRBTEST.COM'} at 127.0.0.1:54054
I20260927 10:55:07.408823 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 846bef9fa183401d8842abcd7fe9c8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:07 UTC) from {username='oryx', principal='oryx/127.1.46.253@KRBTEST.COM'} at 127.0.0.1:38790
I20260927 10:55:07.408951 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 846bef9fa183401d8842abcd7fe9c8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:07 UTC) from {username='oryx', principal='oryx/127.1.46.253@KRBTEST.COM'} at 127.0.0.1:44312
I20260927 10:55:07.409175 8666 tablet_replica.cc:330] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: stopping tablet replica
I20260927 10:55:07.409214 8652 tablet_replica.cc:330] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: stopping tablet replica
I20260927 10:55:07.409348 8652 raft_consensus.cc:2229] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:07.409396 8666 raft_consensus.cc:2229] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:07.409495 8652 raft_consensus.cc:2258] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:07.409780 8666 raft_consensus.cc:2258] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:07.409798 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 8048
I20260927 10:55:07.410225 8652 ts_tablet_manager.cc:1905] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:07.411015 8666 ts_tablet_manager.cc:1905] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:07.413367 8652 ts_tablet_manager.cc:1918] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:07.413451 8652 log.cc:1196] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/846bef9fa183401d8842abcd7fe9c8ea
I20260927 10:55:07.413672 8666 ts_tablet_manager.cc:1918] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:07.413766 8652 ts_tablet_manager.cc:1939] T 846bef9fa183401d8842abcd7fe9c8ea P 53a1dac39e57495fad5fc4a00e2e1eed: Deleting consensus metadata
I20260927 10:55:07.413815 8666 log.cc:1196] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/846bef9fa183401d8842abcd7fe9c8ea
I20260927 10:55:07.414193 8666 ts_tablet_manager.cc:1939] T 846bef9fa183401d8842abcd7fe9c8ea P 916eb4920b88444aa424619c328e0fb4: Deleting consensus metadata
I20260927 10:55:07.414379 7888 catalog_manager.cc:4841] TS 53a1dac39e57495fad5fc4a00e2e1eed (127.1.46.195:39857): tablet 846bef9fa183401d8842abcd7fe9c8ea (table test-table [id=90edf74209ab4231b130feefdb56ad65]) successfully deleted
I20260927 10:55:07.415074 7889 catalog_manager.cc:4841] TS 916eb4920b88444aa424619c328e0fb4 (127.1.46.194:37281): tablet 846bef9fa183401d8842abcd7fe9c8ea (table test-table [id=90edf74209ab4231b130feefdb56ad65]) successfully deleted
I20260927 10:55:07.420198 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 8282
I20260927 10:55:07.420948 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:07.420281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33931 (local address 127.0.0.1:44322)
0927 10:55:07.420841 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:07.420887 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:33931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
W20260927 10:55:07.421116 7890 catalog_manager.cc:4584] TS c52462feee3e45bf8238de90cf56aaa5 (127.1.46.193:33931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 846bef9fa183401d8842abcd7fe9c8ea: Network error: Client connection negotiation failed: client connection to 127.1.46.193:33931: connect: Connection refused (error 111)
I20260927 10:55:07.426373 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 8418
I20260927 10:55:07.431699 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 7768
I20260927 10:55:07.437431 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 7876
2026-09-27T10:55:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5374 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:07 dist-test-slave-cffh krb5kdc[8670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:07 dist-test-slave-cffh krb5kdc[8670](info): set up 2 sockets
Sep 27 10:55:07 dist-test-slave-cffh krb5kdc[8670](info): commencing operation
krb5kdc: starting...
W20260927 10:55:08.483193 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:55:08 dist-test-slave-cffh krb5kdc[8670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab.
Entry for principal oryx/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab.
I20260927 10:55:08.600989 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44715
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab
--principal=oryx/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:44715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:08.707923 8683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:08.708194 8683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:08.708266 8683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:08.712142 8683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:08.712222 8683 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:55:08.712250 8683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:08.712307 8683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:08.712347 8683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:08.712393 8683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:08.717242 8683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:44715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.254.keytab
--principal=oryx/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:08.718220 8683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:08.719185 8683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:08.727447 8691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:08.727413 8688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:08.727521 8683 server_base.cc:1034] running on GCE node
W20260927 10:55:08.727451 8689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:08.728392 8683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:08.729496 8683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:08.730654 8683 hybrid_clock.cc:648] HybridClock initialized: now 1790506508730634 us; error 45 us; skew 500 ppm
Sep 27 10:55:08 dist-test-slave-cffh krb5kdc[8670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506508, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:08.736240 8683 init.cc:377] Logged in from keytab as oryx/127.1.46.254@KRBTEST.COM (short username oryx)
I20260927 10:55:08.737012 8683 webserver.cc:460] Webserver started at http://127.1.46.254:43411/ using document root <none> and password file <none>
I20260927 10:55:08.737509 8683 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:08.737563 8683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:08.737711 8683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:08.739614 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "47e20a38834e4d488c644002c9b2320d"
format_stamp: "Formatted at 2026-09-27 10:55:08 on dist-test-slave-cffh"
server_key: "948fd50b32d794e072fc611ac2dc67a1"
server_key_iv: "8769585e9908b5bad741a202ef4c1614"
server_key_version: "encryptionkey@0"
I20260927 10:55:08.740095 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "47e20a38834e4d488c644002c9b2320d"
format_stamp: "Formatted at 2026-09-27 10:55:08 on dist-test-slave-cffh"
server_key: "948fd50b32d794e072fc611ac2dc67a1"
server_key_iv: "8769585e9908b5bad741a202ef4c1614"
server_key_version: "encryptionkey@0"
I20260927 10:55:08.743592 8683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:08.745828 8698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:08.747160 8683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:08.747278 8683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "47e20a38834e4d488c644002c9b2320d"
format_stamp: "Formatted at 2026-09-27 10:55:08 on dist-test-slave-cffh"
server_key: "948fd50b32d794e072fc611ac2dc67a1"
server_key_iv: "8769585e9908b5bad741a202ef4c1614"
server_key_version: "encryptionkey@0"
I20260927 10:55:08.747375 8683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:08.769027 8683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:08.773685 8683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:08.773891 8683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:08.786367 8683 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:44715
I20260927 10:55:08.786429 8760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:44715 every 8 connection(s)
I20260927 10:55:08.787762 8683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:08.790774 8761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:08.796787 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 8683
I20260927 10:55:08.796921 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:55:08.797264 1211 external_mini_cluster.cc:1428] Setting key bea5ff2118fdbeca58d64b30e8f64d8b
I20260927 10:55:08.797279 8761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Bootstrap starting.
I20260927 10:55:08.800118 8761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:08.801169 8761 log.cc:824] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:08.803452 8761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: No bootstrap required, opened a new log
Sep 27 10:55:08 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:08.809095 8761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e20a38834e4d488c644002c9b2320d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44715 } }
I20260927 10:55:08.809417 8761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:08.809501 8761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e20a38834e4d488c644002c9b2320d, State: Initialized, Role: FOLLOWER
I20260927 10:55:08.810002 8761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e20a38834e4d488c644002c9b2320d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44715 } }
I20260927 10:55:08.810184 8761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:08.810258 8761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:08.810379 8761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:08.811612 8761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e20a38834e4d488c644002c9b2320d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44715 } }
I20260927 10:55:08.811990 8761 leader_election.cc:302] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47e20a38834e4d488c644002c9b2320d; no voters:
I20260927 10:55:08.812343 8761 leader_election.cc:288] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:08.812427 8766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:08.812654 8766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [term 1 LEADER]: Becoming Leader. State: Replica: 47e20a38834e4d488c644002c9b2320d, State: Running, Role: LEADER
I20260927 10:55:08.812974 8766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e20a38834e4d488c644002c9b2320d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44715 } }
I20260927 10:55:08.813726 8761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:08.814612 8767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e20a38834e4d488c644002c9b2320d. Latest consensus state: current_term: 1 leader_uuid: "47e20a38834e4d488c644002c9b2320d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e20a38834e4d488c644002c9b2320d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44715 } } }
I20260927 10:55:08.814726 8767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:08.816156 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:08.798725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.1.46.254:44715)
0927 10:55:08.799205 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:08.799240 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:08.799304 (+ 64us) server_negotiation.cc:407] Connection header received
0927 10:55:08.800262 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:08.800288 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:08.800777 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:08.801193 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:08.802132 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:08.803605 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:08.804822 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:08.805638 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:08.805812 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:08.809292 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:08.809299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:08.809314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:08.809349 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:08.811781 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:08.812621 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:08.812628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:08.812634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:08.812809 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:08.813182 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:08.813186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:08.813188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:08.813655 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:08.813835 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:08.814778 (+ 943us) server_negotiation.cc:299] Negotiation successful
0927 10:55:08.815108 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":153,"threads_started":1}
I20260927 10:55:08.816395 8768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e20a38834e4d488c644002c9b2320d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e20a38834e4d488c644002c9b2320d" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44715 } } }
I20260927 10:55:08.816757 8768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d [sys.catalog]: This master's current role is: LEADER
W20260927 10:55:08.818913 8782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:55:08.818988 8782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:08.819056 8775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:08.820070 8775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:08.825577 8775 catalog_manager.cc:1347] Generated new cluster ID: 1e16a2d7ceb440d6b6e6165861f5551d
I20260927 10:55:08.825636 8775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:08.848575 8775 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:08.849889 8775 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:08.854867 8775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Generated new TSK 0
I20260927 10:55:08.855542 8775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab.
Entry for principal oryx/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab.
I20260927 10:55:08.887632 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44715
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab
--principal=oryx/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:09.002218 8787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:09.002442 8787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:09.002498 8787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:09.006273 8787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:09.006330 8787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:09.006382 8787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:55:09.006448 8787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:09.011389 8787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab
--principal=oryx/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:44715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:09.012279 8787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:09.013206 8787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:09.022202 8793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:09.022248 8795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:09.022217 8792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:09.022426 8787 server_base.cc:1034] running on GCE node
I20260927 10:55:09.022919 8787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:09.023478 8787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:09.024622 8787 hybrid_clock.cc:648] HybridClock initialized: now 1790506509024591 us; error 43 us; skew 500 ppm
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:09.027194 8787 init.cc:377] Logged in from keytab as oryx/127.1.46.193@KRBTEST.COM (short username oryx)
I20260927 10:55:09.027979 8787 webserver.cc:460] Webserver started at http://127.1.46.193:37085/ using document root <none> and password file <none>
I20260927 10:55:09.028492 8787 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:09.028558 8787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:09.028786 8787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:09.030710 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "ed0ff77592124b51b580877f513eecd6"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "56a2f04ddbc884111659113ab277fba8"
server_key_iv: "345e7498af4cc1a8e99727534267eca7"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.031234 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "ed0ff77592124b51b580877f513eecd6"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "56a2f04ddbc884111659113ab277fba8"
server_key_iv: "345e7498af4cc1a8e99727534267eca7"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.034673 8787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:09.036875 8802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:09.037955 8787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:09.038069 8787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "ed0ff77592124b51b580877f513eecd6"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "56a2f04ddbc884111659113ab277fba8"
server_key_iv: "345e7498af4cc1a8e99727534267eca7"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.038163 8787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:09.058796 8787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:09.062085 8787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:09.062280 8787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:09.063681 8787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:09.063726 8787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:09.063789 8787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:09.063819 8787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:09.074255 8787 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:39777
I20260927 10:55:09.074316 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:39777 every 8 connection(s)
I20260927 10:55:09.075170 8787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:09.078329 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.064528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:48817 (local address 127.1.46.254:44715)
0927 10:55:09.064648 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.064652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.065606 (+ 954us) server_negotiation.cc:407] Connection header received
0927 10:55:09.066758 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.066762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.066848 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.066913 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.068677 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.069564 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.071437 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.071667 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.071753 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.074761 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:09.074766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:09.074770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:09.074799 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:09.076479 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.076961 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.076963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.076965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.077051 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.077391 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.077394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.077396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.077602 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:09.077684 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.078113 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.078211 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:55:09.079067 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.065021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44715 (local address 127.1.46.193:48817)
0927 10:55:09.065468 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.065506 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.066460 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.067401 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.067412 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.067822 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:09.068536 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.068556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.069723 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.069727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.071320 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.071329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.071826 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.071830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.072047 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.072692 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:09.072715 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:09.074558 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:09.076576 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.076580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.076590 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.076842 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.077128 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.077130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.077131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.077251 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.077701 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:09.077706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:09.077902 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.078038 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.078255 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":130,"threads_started":1}
I20260927 10:55:09.080224 8918 heartbeater.cc:347] Connected to a master server at 127.1.46.254:44715
I20260927 10:55:09.080355 8918 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:09.080862 8918 heartbeater.cc:511] Master 127.1.46.254:44715 requested a full tablet report, sending...
I20260927 10:55:09.082408 8714 ts_manager.cc:194] Registered new tserver with Master: ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777)
I20260927 10:55:09.083539 8714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:48817
I20260927 10:55:09.084108 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 8787
I20260927 10:55:09.084263 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:09.084559 1211 external_mini_cluster.cc:1428] Setting key 7c88da67f1e2ae3b3c733b10985dd182
I20260927 10:55:09.089751 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.082449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:45967 (local address 127.1.46.254:44715)
0927 10:55:09.082614 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.082618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.082793 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:55:09.082892 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.082895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.082940 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.083022 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.083420 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.084042 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.085160 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.085466 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.085576 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.086668 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:09.086675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:09.086681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:09.086712 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:09.088139 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.088574 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.088580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.088583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.088665 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.088955 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.088959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.088961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.089130 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:09.089233 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.089501 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.089603 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:09.089769 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.082377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44715 (local address 127.1.46.193:45967)
0927 10:55:09.082711 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.082724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.082818 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.083036 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.083039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.083224 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:09.083310 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.083315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.084201 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.084206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.085024 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.085033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.085590 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.085594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.085706 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.086199 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:09.086220 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:09.086508 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:09.088249 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.088256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.088259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.088464 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.088760 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.088765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.088768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.088858 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.089234 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:09.089237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:09.089338 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.089441 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.089590 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for oryx/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab.
Entry for principal oryx/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab.
I20260927 10:55:09.111003 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44715
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab
--principal=oryx/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:09.218169 8927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:09.218444 8927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:09.218518 8927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:09.222430 8927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:09.222512 8927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:09.222637 8927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:55:09.222720 8927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:09.227869 8927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab
--principal=oryx/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:44715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:09.228786 8927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:09.229715 8927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:09.238721 8932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:09.238731 8933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:09.238816 8927 server_base.cc:1034] running on GCE node
W20260927 10:55:09.238731 8935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:09.239346 8927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:09.239907 8927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:09.241094 8927 hybrid_clock.cc:648] HybridClock initialized: now 1790506509241084 us; error 38 us; skew 500 ppm
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:09.243741 8927 init.cc:377] Logged in from keytab as oryx/127.1.46.194@KRBTEST.COM (short username oryx)
I20260927 10:55:09.244592 8927 webserver.cc:460] Webserver started at http://127.1.46.194:37577/ using document root <none> and password file <none>
I20260927 10:55:09.245119 8927 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:09.245203 8927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:09.245415 8927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:09.247496 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "c9e860429f2f4a95adb75369bac242ae"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "36e8cbc0d620a288642077b5da81e9d8"
server_key_iv: "689bf15afac20a622543700bc71afafc"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.248056 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "c9e860429f2f4a95adb75369bac242ae"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "36e8cbc0d620a288642077b5da81e9d8"
server_key_iv: "689bf15afac20a622543700bc71afafc"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.251448 8927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:09.253793 8942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:09.255026 8927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:09.255153 8927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "c9e860429f2f4a95adb75369bac242ae"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "36e8cbc0d620a288642077b5da81e9d8"
server_key_iv: "689bf15afac20a622543700bc71afafc"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.255267 8927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:09.266567 8927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:09.269194 8927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:09.269377 8927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:09.270740 8927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:09.270807 8927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:09.270866 8927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:09.270918 8927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:09.281481 8927 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:39739
I20260927 10:55:09.281569 9057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:39739 every 8 connection(s)
I20260927 10:55:09.282475 8927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:09.284840 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.271467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:33979 (local address 127.1.46.254:44715)
0927 10:55:09.271616 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.271620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.272486 (+ 866us) server_negotiation.cc:407] Connection header received
0927 10:55:09.273721 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.273727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.273793 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.273856 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.275622 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.276379 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.277812 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.278058 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.278174 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.281232 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:09.281236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:09.281240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:09.281272 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:09.282793 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.283366 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.283372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.283375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.283465 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.283803 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.283808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.283811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.283991 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:09.284105 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.284559 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.284676 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:55:09.285626 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.271833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44715 (local address 127.1.46.194:33979)
0927 10:55:09.272310 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.272376 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.273461 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.274290 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.274300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.274749 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:09.275438 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.275459 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.276515 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.276519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.277693 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.277702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.278217 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.278221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.278378 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.279446 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:09.279461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:09.281082 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:09.282900 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.282907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.282920 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.283275 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.283568 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.283572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.283575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.283706 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.284102 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:09.284108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:09.284340 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.284518 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.284814 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260927 10:55:09.286458 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 8927
I20260927 10:55:09.286602 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:09.286914 1211 external_mini_cluster.cc:1428] Setting key 1cc2e1eafc0a88a24e0a5d9ff0abc3f2
I20260927 10:55:09.289206 9058 heartbeater.cc:347] Connected to a master server at 127.1.46.254:44715
I20260927 10:55:09.289364 9058 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:09.289844 9058 heartbeater.cc:511] Master 127.1.46.254:44715 requested a full tablet report, sending...
I20260927 10:55:09.290874 8715 ts_manager.cc:194] Registered new tserver with Master: c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739)
I20260927 10:55:09.291661 8715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:33979
WARNING: no policy specified for oryx/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:55:09.297802 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.289442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:50501 (local address 127.1.46.254:44715)
0927 10:55:09.289579 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.289585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.290142 (+ 557us) server_negotiation.cc:407] Connection header received
0927 10:55:09.290234 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.290237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.290285 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.290376 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.290905 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.291722 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.293027 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.293289 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.293403 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.294527 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:09.294531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:09.294534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:09.294558 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:09.296072 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.296551 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.296555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.296558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.296647 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.296926 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.296931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.296933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.297141 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:09.297263 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.297539 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.297654 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:55:09.297811 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.289458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44715 (local address 127.1.46.194:50501)
0927 10:55:09.290032 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.290057 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.290158 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.290400 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.290404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.290643 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:09.290769 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.290777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.291892 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.291895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.292826 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.292835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.293405 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.293409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.293509 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.294078 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:09.294091 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:09.294388 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:09.296203 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.296208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.296211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.296446 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.296747 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.296751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.296754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.296838 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.297279 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:09.297284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:09.297375 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.297478 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.297613 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab.
Entry for principal oryx/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab.
I20260927 10:55:09.314095 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:44715
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab
--principal=oryx/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:09.425295 9067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:09.425545 9067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:09.425588 9067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:09.429536 9067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:09.429593 9067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:09.429632 9067 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:55:09.429733 9067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:09.434715 9067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab
--principal=oryx/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.46.254:44715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:09.435722 9067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:09.436622 9067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:09.445905 9072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:09.445957 9075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:09.446079 9067 server_base.cc:1034] running on GCE node
W20260927 10:55:09.446130 9073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:09.446679 9067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:09.447265 9067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:09.448467 9067 hybrid_clock.cc:648] HybridClock initialized: now 1790506509448453 us; error 29 us; skew 500 ppm
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:09.451305 9067 init.cc:377] Logged in from keytab as oryx/127.1.46.195@KRBTEST.COM (short username oryx)
I20260927 10:55:09.452131 9067 webserver.cc:460] Webserver started at http://127.1.46.195:43879/ using document root <none> and password file <none>
I20260927 10:55:09.452656 9067 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:09.452745 9067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:09.452967 9067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:09.454974 9067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "5a0a16b4348244fcb02f48add05a6d8a"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "6ff1b10f9de0292f68f4be86610e86a2"
server_key_iv: "fa2a38de698e0603f8d844fb9be63fcc"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.455519 9067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "5a0a16b4348244fcb02f48add05a6d8a"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "6ff1b10f9de0292f68f4be86610e86a2"
server_key_iv: "fa2a38de698e0603f8d844fb9be63fcc"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.459057 9067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:09.461298 9082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:09.462433 9067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:09.462561 9067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "5a0a16b4348244fcb02f48add05a6d8a"
format_stamp: "Formatted at 2026-09-27 10:55:09 on dist-test-slave-cffh"
server_key: "6ff1b10f9de0292f68f4be86610e86a2"
server_key_iv: "fa2a38de698e0603f8d844fb9be63fcc"
server_key_version: "encryptionkey@0"
I20260927 10:55:09.462710 9067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:09.474725 9067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:09.477831 9067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:09.478018 9067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:09.479489 9067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:09.479560 9067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:09.479631 9067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:09.479674 9067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:09.491037 9067 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:44207
I20260927 10:55:09.491129 9197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:44207 every 8 connection(s)
I20260927 10:55:09.492031 9067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:09.494405 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.480357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:55437 (local address 127.1.46.254:44715)
0927 10:55:09.480483 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.480486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.481523 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 10:55:09.482903 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.482907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.482974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.483076 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.484645 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.485537 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.487292 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.487698 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.487801 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.490734 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:09.490738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:09.490742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:09.490774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:09.492490 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.493128 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.493131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.493132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.493210 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.493492 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.493493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.493495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.493647 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:09.493716 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.494194 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.494289 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:55:09.495234 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.480880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44715 (local address 127.1.46.195:55437)
0927 10:55:09.481333 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.481399 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.482655 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.483380 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.483390 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.483854 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:09.484470 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.484493 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.485681 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.485685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.487170 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.487182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.487841 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.487845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.488039 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.488615 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:09.488630 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:09.490503 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:09.492639 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.492645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.492659 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.492990 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.493281 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.493284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.493286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.493416 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.493734 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:09.493738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:09.493961 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.494128 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.494371 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
I20260927 10:55:09.496523 9198 heartbeater.cc:347] Connected to a master server at 127.1.46.254:44715
I20260927 10:55:09.496678 9198 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:09.497187 9198 heartbeater.cc:511] Master 127.1.46.254:44715 requested a full tablet report, sending...
I20260927 10:55:09.498137 8715 ts_manager.cc:194] Registered new tserver with Master: 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207)
I20260927 10:55:09.498960 8715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:55437
I20260927 10:55:09.500324 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 9067
I20260927 10:55:09.500442 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:09.500730 1211 external_mini_cluster.cc:1428] Setting key 45db9b25b7ca030542de94ac4b24ac88
I20260927 10:55:09.502786 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:55:09.505527 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.498481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:33625 (local address 127.1.46.254:44715)
0927 10:55:09.498643 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.498646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.498800 (+ 154us) server_negotiation.cc:407] Connection header received
0927 10:55:09.498895 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.498897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.498930 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.498995 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.499460 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.499950 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.500871 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.501070 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.501180 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.502369 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:09.502372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:09.502374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:09.502394 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:09.503851 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.504332 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.504338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.504342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.504439 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.504744 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.504747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.504748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.504907 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:09.504982 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.505310 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.505407 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:55:09.505609 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.498402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44715 (local address 127.1.46.195:33625)
0927 10:55:09.498718 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.498732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.498821 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.499039 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.499044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.499249 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:09.499347 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.499352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.500081 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.500085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.500732 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.500740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.501180 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.501184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.501311 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.501810 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:09.501825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:09.502233 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:09.503980 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.503984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.503987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.504178 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.504539 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:09.504543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:09.504547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:09.504647 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:09.505000 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:09.505004 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:09.505102 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.505217 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.505378 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":86,"threads_started":1}
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:55:09.526041 8764 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:09.520405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.1.46.254:44715)
0927 10:55:09.520524 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.520527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.520539 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:09.520575 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.520577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.520614 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.520691 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.521335 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.521826 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.522648 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.522868 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.522944 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.525896 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47160: BlockingRecv error: recv got EOF from 127.0.0.1:47160 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:55:09 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506509, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:09.534842 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.527684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.1.46.254:44715)
0927 10:55:09.527789 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.527792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.527895 (+ 103us) server_negotiation.cc:407] Connection header received
0927 10:55:09.527989 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.527991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.528024 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.528083 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:09.528587 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.529047 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.529783 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.529973 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.530043 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.532248 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:09.532251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:09.532254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:09.532274 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:09.533495 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.533866 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.533868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.533869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.533936 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:09.534156 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:09.534158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:09.534159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:09.534306 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:09.534402 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.534653 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.534735 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:55:09.537647 8715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:09.540701 8715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:09.555150 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.549924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.1.46.193:39777)
0927 10:55:09.550292 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.550307 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.550742 (+ 435us) server_negotiation.cc:407] Connection header received
0927 10:55:09.550828 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.550836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.551018 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.551193 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:09.551405 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.552266 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.553785 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.554434 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.554541 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.554749 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.554896 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.554990 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":72,"threads_started":1}
I20260927 10:55:09.555159 9220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.549421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39777 (local address 127.0.0.1:44156)
0927 10:55:09.550626 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.550640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.550733 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.551190 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.551192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.551212 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:09.551301 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.551306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.552382 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.552386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.553666 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.553675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.554562 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.554586 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.554696 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.554752 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.554859 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.555018 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":195,"thread_start_us":100,"threads_started":1}
I20260927 10:55:09.555436 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.549215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44207 (local address 127.0.0.1:48112)
0927 10:55:09.550053 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.550102 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.550297 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.550892 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.550897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.550917 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:09.551033 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.551039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.552403 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.552407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.553835 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.553845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.555053 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.555057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.555149 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.555173 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.555245 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.555327 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20260927 10:55:09.555752 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.549395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.1.46.195:44207)
0927 10:55:09.549695 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.549706 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.550188 (+ 482us) server_negotiation.cc:407] Connection header received
0927 10:55:09.550457 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.550465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.550727 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.550882 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:09.551164 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.552271 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.553962 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.554683 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.554780 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.554942 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.555517 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.555609 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
I20260927 10:55:09.556250 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.550274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.1.46.194:39739)
0927 10:55:09.550628 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.550640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.550881 (+ 241us) server_negotiation.cc:407] Connection header received
0927 10:55:09.550950 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.550954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.551193 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.551322 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:09.551701 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.552801 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.554227 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.555052 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.555179 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.555334 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.556024 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.556119 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":106,"threads_started":1}
I20260927 10:55:09.558163 9132 tablet_service.cc:1467] Processing CreateTablet for tablet 881783a971e245b5b28ab21efddd1375 (DEFAULT_TABLE table=test-table [id=b16adac481cf483ab820ff569b783845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:09.558212 8852 tablet_service.cc:1467] Processing CreateTablet for tablet 881783a971e245b5b28ab21efddd1375 (DEFAULT_TABLE table=test-table [id=b16adac481cf483ab820ff569b783845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:09.559152 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881783a971e245b5b28ab21efddd1375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:09.559145 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881783a971e245b5b28ab21efddd1375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:09.564616 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.549887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:39739 (local address 127.0.0.1:59314)
0927 10:55:09.550659 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.550677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.550793 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.551462 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.551467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.551486 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:09.551559 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.551565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.552927 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.552930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.554091 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.554099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.555395 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.555400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.555473 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.555490 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.555581 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.564424 (+ 8843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":50,"thread_start_us":75,"threads_started":1}
I20260927 10:55:09.565785 9227 tablet_bootstrap.cc:492] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Bootstrap starting.
I20260927 10:55:09.566324 9226 tablet_bootstrap.cc:492] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Bootstrap starting.
I20260927 10:55:09.567479 8992 tablet_service.cc:1467] Processing CreateTablet for tablet 881783a971e245b5b28ab21efddd1375 (DEFAULT_TABLE table=test-table [id=b16adac481cf483ab820ff569b783845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:09.567981 9227 tablet_bootstrap.cc:654] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:09.568363 9226 tablet_bootstrap.cc:654] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:09.568513 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881783a971e245b5b28ab21efddd1375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:09.568819 9227 log.cc:824] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:09.569156 9226 log.cc:824] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:09.570756 9227 tablet_bootstrap.cc:492] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: No bootstrap required, opened a new log
I20260927 10:55:09.570771 9226 tablet_bootstrap.cc:492] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: No bootstrap required, opened a new log
I20260927 10:55:09.570946 9226 ts_tablet_manager.cc:1397] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:09.571053 9227 ts_tablet_manager.cc:1397] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260927 10:55:09.573709 9226 raft_consensus.cc:348] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.573935 9226 raft_consensus.cc:374] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:09.573985 9226 raft_consensus.cc:729] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0a16b4348244fcb02f48add05a6d8a, State: Initialized, Role: FOLLOWER
I20260927 10:55:09.574384 9226 consensus_queue.cc:260] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.574990 9198 heartbeater.cc:503] Master 127.1.46.254:44715 was elected leader, sending a full tablet report...
I20260927 10:55:09.574918 9227 raft_consensus.cc:348] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.575152 9227 raft_consensus.cc:374] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:09.575209 9227 raft_consensus.cc:729] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0ff77592124b51b580877f513eecd6, State: Initialized, Role: FOLLOWER
I20260927 10:55:09.575613 9227 consensus_queue.cc:260] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.576090 9231 tablet_bootstrap.cc:492] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Bootstrap starting.
I20260927 10:55:09.576833 9227 ts_tablet_manager.cc:1428] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:09.577184 8918 heartbeater.cc:503] Master 127.1.46.254:44715 was elected leader, sending a full tablet report...
I20260927 10:55:09.578435 9231 tablet_bootstrap.cc:654] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:09.578629 9226 ts_tablet_manager.cc:1428] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260927 10:55:09.579275 9231 log.cc:824] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:09.581101 9231 tablet_bootstrap.cc:492] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: No bootstrap required, opened a new log
I20260927 10:55:09.581327 9231 ts_tablet_manager.cc:1397] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:09.584111 9231 raft_consensus.cc:348] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.584359 9231 raft_consensus.cc:374] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:09.584439 9231 raft_consensus.cc:729] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e860429f2f4a95adb75369bac242ae, State: Initialized, Role: FOLLOWER
I20260927 10:55:09.584829 9231 consensus_queue.cc:260] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.585424 9058 heartbeater.cc:503] Master 127.1.46.254:44715 was elected leader, sending a full tablet report...
I20260927 10:55:09.585635 9231 ts_tablet_manager.cc:1428] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 10:55:09.742926 9199 tablet.cc:2369] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:09.783521 9234 raft_consensus.cc:482] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 10:55:09.783730 9059 tablet.cc:2369] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:09.783761 9234 raft_consensus.cc:504] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.784987 9234 leader_election.cc:288] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207), ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777)
I20260927 10:55:09.789361 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.785327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:52987 (local address 127.1.46.195:44207)
0927 10:55:09.785449 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.785456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.785471 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:55:09.785526 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.785528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.785589 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.785687 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:09.785953 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.786776 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.788269 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.788773 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.788917 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.788971 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.789155 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.789221 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:09.789361 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.785178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44207 (local address 127.1.46.194:52987)
0927 10:55:09.785296 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.785315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.785403 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.785697 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.785699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.785712 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:09.785848 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.785852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.786883 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.786886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.788134 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.788144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.788902 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.788905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.789007 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.789036 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.789136 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.789204 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 10:55:09.789872 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.785436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:40835 (local address 127.1.46.193:39777)
0927 10:55:09.785540 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.785544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.785672 (+ 128us) server_negotiation.cc:407] Connection header received
0927 10:55:09.785806 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.785816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.785857 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.785968 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:09.786274 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.786990 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.788589 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.789199 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.789342 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.789387 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.789692 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.789760 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:55:09.790057 9136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881783a971e245b5b28ab21efddd1375" candidate_uuid: "c9e860429f2f4a95adb75369bac242ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0a16b4348244fcb02f48add05a6d8a" is_pre_election: true
I20260927 10:55:09.790239 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.785252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39777 (local address 127.1.46.194:40835)
0927 10:55:09.785578 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.785590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.785701 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.785971 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.785973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.785992 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:09.786134 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.786141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.787095 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.787101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.788457 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.788466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.789468 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.789472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.789566 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.789587 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.790008 (+ 421us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.790100 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":72,"threads_started":1}
I20260927 10:55:09.790354 9136 raft_consensus.cc:2454] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e860429f2f4a95adb75369bac242ae in term 0.
I20260927 10:55:09.790789 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881783a971e245b5b28ab21efddd1375" candidate_uuid: "c9e860429f2f4a95adb75369bac242ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0ff77592124b51b580877f513eecd6" is_pre_election: true
I20260927 10:55:09.790889 8945 leader_election.cc:302] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, c9e860429f2f4a95adb75369bac242ae; no voters:
I20260927 10:55:09.791088 8872 raft_consensus.cc:2454] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e860429f2f4a95adb75369bac242ae in term 0.
I20260927 10:55:09.791144 9234 raft_consensus.cc:2781] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:09.791221 9234 raft_consensus.cc:482] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:09.791271 9234 raft_consensus.cc:3037] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:09.792358 9234 raft_consensus.cc:504] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.792724 9234 leader_election.cc:288] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [CANDIDATE]: Term 1 election: Requested vote from peers 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207), ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777)
I20260927 10:55:09.793040 9136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881783a971e245b5b28ab21efddd1375" candidate_uuid: "c9e860429f2f4a95adb75369bac242ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0a16b4348244fcb02f48add05a6d8a"
I20260927 10:55:09.793151 9136 raft_consensus.cc:3037] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:09.794533 9136 raft_consensus.cc:2454] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e860429f2f4a95adb75369bac242ae in term 1.
I20260927 10:55:09.794867 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881783a971e245b5b28ab21efddd1375" candidate_uuid: "c9e860429f2f4a95adb75369bac242ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0ff77592124b51b580877f513eecd6"
I20260927 10:55:09.794885 8945 leader_election.cc:302] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, c9e860429f2f4a95adb75369bac242ae; no voters:
I20260927 10:55:09.795006 8872 raft_consensus.cc:3037] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:09.795063 9234 raft_consensus.cc:2781] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:09.795248 9234 raft_consensus.cc:686] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 1 LEADER]: Becoming Leader. State: Replica: c9e860429f2f4a95adb75369bac242ae, State: Running, Role: LEADER
I20260927 10:55:09.795578 9234 consensus_queue.cc:237] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } }
I20260927 10:55:09.796307 8872 raft_consensus.cc:2454] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e860429f2f4a95adb75369bac242ae in term 1.
I20260927 10:55:09.798825 8715 catalog_manager.cc:5495] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e860429f2f4a95adb75369bac242ae (127.1.46.194). New cstate: current_term: 1 leader_uuid: "c9e860429f2f4a95adb75369bac242ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } health_report { overall_health: HEALTHY } } }
W20260927 10:55:09.826774 8919 tablet.cc:2369] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:09.846591 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.842553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.1.46.194:39739)
0927 10:55:09.842716 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.842721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.842738 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:55:09.842870 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.842873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.842929 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.843070 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:09.843354 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.843918 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.844875 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.845128 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.845249 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.845573 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:09.846018 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:09.846286 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.846360 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.846417 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:55:09.852748 9136 raft_consensus.cc:1261] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Refusing update from remote peer c9e860429f2f4a95adb75369bac242ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:09.852748 8872 raft_consensus.cc:1261] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Refusing update from remote peer c9e860429f2f4a95adb75369bac242ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:09.853410 9236 consensus_queue.cc:1035] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:09.853628 9234 consensus_queue.cc:1035] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:09.862149 9240 mvcc.cc:205] Tried to move back new op lower bound from 7333914664349433856 to 7333914664125919232. Current Snapshot: MvccSnapshot[applied={T|T < 7333914664349433856}]
I20260927 10:55:09.871290 8714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:09.871428 8714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:09.873734 8714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Sending DeleteTablet for 3 replicas of tablet 881783a971e245b5b28ab21efddd1375
I20260927 10:55:09.874543 9132 tablet_service.cc:1514] Processing DeleteTablet for tablet 881783a971e245b5b28ab21efddd1375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:09 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:48112
I20260927 10:55:09.874675 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet 881783a971e245b5b28ab21efddd1375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:09 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:59314
I20260927 10:55:09.875061 8852 tablet_service.cc:1514] Processing DeleteTablet for tablet 881783a971e245b5b28ab21efddd1375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:09 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44156
I20260927 10:55:09.875494 9251 tablet_replica.cc:330] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: stopping tablet replica
I20260927 10:55:09.875801 9251 raft_consensus.cc:2229] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:09.875764 8714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:55:09.876096 9251 raft_consensus.cc:2258] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:55:09.876147 8714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:09.876945 9249 tablet_replica.cc:330] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: stopping tablet replica
I20260927 10:55:09.877220 9249 raft_consensus.cc:2229] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:09.877523 9249 raft_consensus.cc:2258] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:09.877791 9251 ts_tablet_manager.cc:1905] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:09.878003 9250 tablet_replica.cc:330] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: stopping tablet replica
I20260927 10:55:09.878273 9250 raft_consensus.cc:2229] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:09.879529 9250 raft_consensus.cc:2258] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:09.880752 9251 ts_tablet_manager.cc:1918] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:09.880928 9251 log.cc:1196] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/881783a971e245b5b28ab21efddd1375
I20260927 10:55:09.881349 9251 ts_tablet_manager.cc:1939] T 881783a971e245b5b28ab21efddd1375 P ed0ff77592124b51b580877f513eecd6: Deleting consensus metadata
I20260927 10:55:09.881678 9250 ts_tablet_manager.cc:1905] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:09.882161 8852 tablet_service.cc:1467] Processing CreateTablet for tablet 2c11c593214c4524acce6bfba5df6596 (DEFAULT_TABLE table=test-table [id=4f7ecb62eec147e2a82cf670175c1933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:09.882467 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c11c593214c4524acce6bfba5df6596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:09.884733 9227 tablet_bootstrap.cc:492] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: Bootstrap starting.
I20260927 10:55:09.884896 8992 tablet_service.cc:1467] Processing CreateTablet for tablet 2c11c593214c4524acce6bfba5df6596 (DEFAULT_TABLE table=test-table [id=4f7ecb62eec147e2a82cf670175c1933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:09.884928 9250 ts_tablet_manager.cc:1918] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:09.885066 9250 log.cc:1196] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/881783a971e245b5b28ab21efddd1375
I20260927 10:55:09.885164 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c11c593214c4524acce6bfba5df6596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:09.885387 9250 ts_tablet_manager.cc:1939] T 881783a971e245b5b28ab21efddd1375 P c9e860429f2f4a95adb75369bac242ae: Deleting consensus metadata
I20260927 10:55:09.885627 9227 tablet_bootstrap.cc:654] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:09.886430 8699 catalog_manager.cc:4841] TS c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739): tablet 881783a971e245b5b28ab21efddd1375 (table test-table [id=b16adac481cf483ab820ff569b783845]) successfully deleted
I20260927 10:55:09.886832 9132 tablet_service.cc:1467] Processing CreateTablet for tablet 2c11c593214c4524acce6bfba5df6596 (DEFAULT_TABLE table=test-table [id=4f7ecb62eec147e2a82cf670175c1933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:09.887108 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c11c593214c4524acce6bfba5df6596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:09.887521 9227 tablet_bootstrap.cc:492] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: No bootstrap required, opened a new log
I20260927 10:55:09.887625 9227 ts_tablet_manager.cc:1397] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:55:09.888325 9227 raft_consensus.cc:348] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.888441 9227 raft_consensus.cc:374] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:09.888474 9227 raft_consensus.cc:729] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0ff77592124b51b580877f513eecd6, State: Initialized, Role: FOLLOWER
I20260927 10:55:09.888588 9227 consensus_queue.cc:260] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.888820 9227 ts_tablet_manager.cc:1428] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:09.889333 9226 tablet_bootstrap.cc:492] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: Bootstrap starting.
I20260927 10:55:09.889657 8699 catalog_manager.cc:4841] TS ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777): tablet 881783a971e245b5b28ab21efddd1375 (table test-table [id=b16adac481cf483ab820ff569b783845]) successfully deleted
I20260927 10:55:09.890341 9226 tablet_bootstrap.cc:654] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:09.890779 9231 tablet_bootstrap.cc:492] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: Bootstrap starting.
I20260927 10:55:09.891747 9231 tablet_bootstrap.cc:654] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:09.891767 9226 tablet_bootstrap.cc:492] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: No bootstrap required, opened a new log
I20260927 10:55:09.891839 9226 ts_tablet_manager.cc:1397] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:55:09.892262 9226 raft_consensus.cc:348] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.892369 9226 raft_consensus.cc:374] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:09.892406 9226 raft_consensus.cc:729] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0a16b4348244fcb02f48add05a6d8a, State: Initialized, Role: FOLLOWER
I20260927 10:55:09.892521 9226 consensus_queue.cc:260] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.892766 9226 ts_tablet_manager.cc:1428] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:09.893077 9231 tablet_bootstrap.cc:492] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: No bootstrap required, opened a new log
I20260927 10:55:09.893154 9231 ts_tablet_manager.cc:1397] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:09.894158 9231 raft_consensus.cc:348] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.894263 9231 raft_consensus.cc:374] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:09.894294 9231 raft_consensus.cc:729] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e860429f2f4a95adb75369bac242ae, State: Initialized, Role: FOLLOWER
I20260927 10:55:09.894407 9231 consensus_queue.cc:260] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.894649 9231 ts_tablet_manager.cc:1428] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:09.894658 9249 ts_tablet_manager.cc:1905] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:09.897121 9249 ts_tablet_manager.cc:1918] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:09.897230 9249 log.cc:1196] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/881783a971e245b5b28ab21efddd1375
I20260927 10:55:09.897528 9249 ts_tablet_manager.cc:1939] T 881783a971e245b5b28ab21efddd1375 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting consensus metadata
I20260927 10:55:09.898463 8701 catalog_manager.cc:4841] TS 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207): tablet 881783a971e245b5b28ab21efddd1375 (table test-table [id=b16adac481cf483ab820ff569b783845]) successfully deleted
I20260927 10:55:09.969553 9230 raft_consensus.cc:482] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:09.969750 9230 raft_consensus.cc:504] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.970871 9230 leader_election.cc:288] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777), c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739)
I20260927 10:55:09.975435 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.971585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:54367 (local address 127.1.46.194:39739)
0927 10:55:09.971733 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.971738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.971815 (+ 77us) server_negotiation.cc:407] Connection header received
0927 10:55:09.971936 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.971939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.971989 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.972091 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:09.972395 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.973109 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.974204 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.974838 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.974972 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.975052 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.975238 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.975311 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:09.975631 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.971217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:51343 (local address 127.1.46.193:39777)
0927 10:55:09.971358 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:09.971363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:09.971379 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:55:09.971568 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:09.971572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:09.971628 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:09.971709 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:09.972169 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.972959 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.974428 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.975066 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.975181 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.975235 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:09.975441 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 10:55:09.975509 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:55:09.975643 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.971109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39777 (local address 127.1.46.195:51343)
0927 10:55:09.971272 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.971290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.971397 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.971852 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.971856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.971897 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:09.972028 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.972038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.973086 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.973089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.974266 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.974275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.975169 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.975173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.975278 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.975303 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.975423 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.975497 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 10:55:09.975988 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:09.971366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:39739 (local address 127.1.46.195:54367)
0927 10:55:09.971700 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:09.971733 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:09.971837 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:09.972120 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:09.972123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:09.972141 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:09.972265 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.972274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.973228 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.973234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.974073 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:09.974082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:09.974963 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:09.974967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:09.975069 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:09.975095 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:55:09.975559 (+ 464us) client_negotiation.cc:240] Negotiation successful
0927 10:55:09.975635 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260927 10:55:09.976112 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11c593214c4524acce6bfba5df6596" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0ff77592124b51b580877f513eecd6" is_pre_election: true
I20260927 10:55:09.976287 8872 raft_consensus.cc:2454] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 0.
I20260927 10:55:09.976464 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11c593214c4524acce6bfba5df6596" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae" is_pre_election: true
I20260927 10:55:09.976620 9083 leader_election.cc:302] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:09.976682 9012 raft_consensus.cc:2454] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 0.
I20260927 10:55:09.976917 9230 raft_consensus.cc:2781] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:09.976980 9230 raft_consensus.cc:482] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:09.977036 9230 raft_consensus.cc:3037] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:09.978412 9230 raft_consensus.cc:504] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.979000 9230 leader_election.cc:288] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 election: Requested vote from peers ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777), c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739)
I20260927 10:55:09.979390 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11c593214c4524acce6bfba5df6596" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0ff77592124b51b580877f513eecd6"
I20260927 10:55:09.979391 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11c593214c4524acce6bfba5df6596" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae"
I20260927 10:55:09.979524 9012 raft_consensus.cc:3037] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:09.979529 8872 raft_consensus.cc:3037] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:09.980566 8872 raft_consensus.cc:2454] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 1.
I20260927 10:55:09.980736 9012 raft_consensus.cc:2454] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 1.
I20260927 10:55:09.980845 9083 leader_election.cc:302] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:09.981027 9230 raft_consensus.cc:2781] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:09.981212 9230 raft_consensus.cc:686] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 LEADER]: Becoming Leader. State: Replica: 5a0a16b4348244fcb02f48add05a6d8a, State: Running, Role: LEADER
I20260927 10:55:09.981451 9230 consensus_queue.cc:237] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:09.984054 8714 catalog_manager.cc:5495] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195). New cstate: current_term: 1 leader_uuid: "5a0a16b4348244fcb02f48add05a6d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } health_report { overall_health: HEALTHY } } }
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506508, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.254@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:10.032480 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.024299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44715 (local address 127.0.0.1:47176)
0927 10:55:10.024570 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:10.024587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:10.024709 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:10.024948 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:10.024951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:10.025416 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:10.025519 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.025526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.026324 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.026328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.027025 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.027034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.027543 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.027546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.027620 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.028132 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:10.028143 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:10.029470 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:10.031177 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:10.031183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:10.031187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:10.031403 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:10.031629 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:10.031631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:10.031632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:10.031704 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:10.032019 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:10.032023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:10.032111 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 10:55:10.032195 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:55:10.032315 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
I20260927 10:55:10.032480 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.024362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.1.46.254:44715)
0927 10:55:10.024497 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.024502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.024665 (+ 163us) server_negotiation.cc:407] Connection header received
0927 10:55:10.024796 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.024798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.024836 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.024905 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:10.025626 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.026174 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.027120 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.027421 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.027542 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.029587 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:10.029590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:10.029591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:10.029611 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:10.031080 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:10.031484 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:10.031486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:10.031488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:10.031556 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:10.031778 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:10.031780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:10.031781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:10.031927 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:10.032011 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.032247 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.032365 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":59}
I20260927 10:55:10.036180 8714 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:47176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:55:10.036965 8714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:10.041455 8852 tablet_service.cc:1467] Processing CreateTablet for tablet 51f8337ba92d475fb2fce25c220f546e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16248c6c3b924a0aa27b3cb9a13eed36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:55:10.041473 8992 tablet_service.cc:1467] Processing CreateTablet for tablet 51f8337ba92d475fb2fce25c220f546e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16248c6c3b924a0aa27b3cb9a13eed36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:55:10.041815 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f8337ba92d475fb2fce25c220f546e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.041839 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f8337ba92d475fb2fce25c220f546e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.041704 9132 tablet_service.cc:1467] Processing CreateTablet for tablet 51f8337ba92d475fb2fce25c220f546e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16248c6c3b924a0aa27b3cb9a13eed36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:55:10.042065 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f8337ba92d475fb2fce25c220f546e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.044528 9231 tablet_bootstrap.cc:492] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: Bootstrap starting.
I20260927 10:55:10.044566 9227 tablet_bootstrap.cc:492] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: Bootstrap starting.
I20260927 10:55:10.044704 9226 tablet_bootstrap.cc:492] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: Bootstrap starting.
I20260927 10:55:10.045334 9227 tablet_bootstrap.cc:654] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.045368 9231 tablet_bootstrap.cc:654] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.045516 9226 tablet_bootstrap.cc:654] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.046361 9227 tablet_bootstrap.cc:492] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: No bootstrap required, opened a new log
I20260927 10:55:10.046453 9227 ts_tablet_manager.cc:1397] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:10.046840 9226 tablet_bootstrap.cc:492] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: No bootstrap required, opened a new log
I20260927 10:55:10.046926 9226 ts_tablet_manager.cc:1397] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:10.046998 9227 raft_consensus.cc:348] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.047113 9227 raft_consensus.cc:374] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.047158 9227 raft_consensus.cc:729] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0ff77592124b51b580877f513eecd6, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.047284 9227 consensus_queue.cc:260] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.047366 9226 raft_consensus.cc:348] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.047492 9226 raft_consensus.cc:374] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.047535 9226 raft_consensus.cc:729] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0a16b4348244fcb02f48add05a6d8a, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.047590 9227 ts_tablet_manager.cc:1428] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.047664 9226 consensus_queue.cc:260] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.047849 9232 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.047899 9226 ts_tablet_manager.cc:1428] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.047986 9232 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:55:10.048079 9230 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.048213 9230 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:55:10.048673 9231 tablet_bootstrap.cc:492] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: No bootstrap required, opened a new log
I20260927 10:55:10.048765 9231 ts_tablet_manager.cc:1397] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 10:55:10.049193 9231 raft_consensus.cc:348] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.049319 9231 raft_consensus.cc:374] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.049358 9231 raft_consensus.cc:729] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e860429f2f4a95adb75369bac242ae, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.049491 9231 consensus_queue.cc:260] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.049759 9231 ts_tablet_manager.cc:1428] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.049938 9234 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.050046 9234 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:55:10.207950 9232 raft_consensus.cc:482] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:10.208084 9232 raft_consensus.cc:504] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.209307 9232 leader_election.cc:288] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739), 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207)
I20260927 10:55:10.213109 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.209570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44207 (local address 127.1.46.193:37979)
0927 10:55:10.209936 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:10.209969 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:10.210077 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:10.210314 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:10.210316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:10.210347 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:10.210457 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.210461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.211177 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.211180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.212021 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.212030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.212666 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.212669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.212766 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.212814 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 10:55:10.212877 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:55:10.212951 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":67,"thread_start_us":105,"threads_started":1}
I20260927 10:55:10.213117 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.209692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:37979 (local address 127.1.46.195:44207)
0927 10:55:10.209808 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.209813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.210074 (+ 261us) server_negotiation.cc:407] Connection header received
0927 10:55:10.210160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.210162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.210201 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.210289 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:10.210607 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.211065 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.212136 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.212539 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.212627 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.212701 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.212911 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.212987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:55:10.213146 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.209586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:37721 (local address 127.1.46.194:39739)
0927 10:55:10.209748 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.209752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.209986 (+ 234us) server_negotiation.cc:407] Connection header received
0927 10:55:10.210112 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.210126 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.210188 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.210245 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:10.210541 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.211060 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.212179 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.212566 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.212696 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.212742 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.212966 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.213030 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:55:10.213140 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.209485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:39739 (local address 127.1.46.193:37721)
0927 10:55:10.209876 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:10.209888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:10.209994 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:10.210284 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:10.210287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:10.210302 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:10.210439 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.210444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.211196 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.211200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.212028 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.212036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.212701 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.212705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.212832 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.212849 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:55:10.212941 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 10:55:10.212999 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":59,"spinlock_wait_cycles":20864,"thread_start_us":111,"threads_started":1}
I20260927 10:55:10.213594 9136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f8337ba92d475fb2fce25c220f546e" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0a16b4348244fcb02f48add05a6d8a" is_pre_election: true
I20260927 10:55:10.213757 9136 raft_consensus.cc:2454] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 0.
I20260927 10:55:10.213829 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f8337ba92d475fb2fce25c220f546e" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae" is_pre_election: true
I20260927 10:55:10.213972 9012 raft_consensus.cc:2454] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 0.
I20260927 10:55:10.214146 8805 leader_election.cc:302] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:10.214404 9232 raft_consensus.cc:2781] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:10.214468 9232 raft_consensus.cc:482] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:10.214510 9232 raft_consensus.cc:3037] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.215552 9232 raft_consensus.cc:504] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.215916 9232 leader_election.cc:288] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 election: Requested vote from peers c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739), 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207)
I20260927 10:55:10.216188 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f8337ba92d475fb2fce25c220f546e" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae"
I20260927 10:55:10.216302 9012 raft_consensus.cc:3037] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.216325 9136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f8337ba92d475fb2fce25c220f546e" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0a16b4348244fcb02f48add05a6d8a"
I20260927 10:55:10.216437 9136 raft_consensus.cc:3037] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.217206 9012 raft_consensus.cc:2454] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 1.
I20260927 10:55:10.217418 9136 raft_consensus.cc:2454] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 1.
I20260927 10:55:10.217464 8803 leader_election.cc:302] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e860429f2f4a95adb75369bac242ae, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:10.217633 9232 raft_consensus.cc:2781] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:10.217828 9232 raft_consensus.cc:686] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [term 1 LEADER]: Becoming Leader. State: Replica: ed0ff77592124b51b580877f513eecd6, State: Running, Role: LEADER
I20260927 10:55:10.218052 9232 consensus_queue.cc:237] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.219165 9269 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: TxnStatusTablet state changed. Reason: New leader ed0ff77592124b51b580877f513eecd6. Latest consensus state: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.219267 9269 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:55:10.219646 9271 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:55:10.219792 9271 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:55:10.220363 8714 catalog_manager.cc:5495] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed0ff77592124b51b580877f513eecd6 (127.1.46.193). New cstate: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } health_report { overall_health: UNKNOWN } } }
I20260927 10:55:10.235384 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.231275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39777 (local address 127.0.0.1:44164)
0927 10:55:10.231531 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:10.231543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:10.231637 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:10.231984 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:10.231986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:10.231993 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:55:10.232075 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.232080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.233111 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.233115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.233772 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.233779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.234307 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.234309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.234371 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.234428 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.235148 (+ 720us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.235161 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 10:55:10.235218 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:55:10.235284 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260927 10:55:10.235396 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.231385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.1.46.193:39777)
0927 10:55:10.231507 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.231514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.231610 (+ 96us) server_negotiation.cc:407] Connection header received
0927 10:55:10.231774 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.231779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.231851 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.231967 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:10.232162 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.232951 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.233876 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.234202 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.234299 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.234552 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.235051 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.235162 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.235230 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.235279 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:55:10.238188 9136 raft_consensus.cc:1261] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Refusing update from remote peer ed0ff77592124b51b580877f513eecd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.238247 9012 raft_consensus.cc:1261] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Refusing update from remote peer ed0ff77592124b51b580877f513eecd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.238744 9269 consensus_queue.cc:1035] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.238914 9232 consensus_queue.cc:1035] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.240296 9234 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: TxnStatusTablet state changed. Reason: New leader ed0ff77592124b51b580877f513eecd6. Latest consensus state: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.240440 9234 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:55:10.241235 9230 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: TxnStatusTablet state changed. Reason: New leader ed0ff77592124b51b580877f513eecd6. Latest consensus state: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.241358 9230 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:55:10.242821 9232 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.242920 9232 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:55:10.243028 9232 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.243096 9232 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P ed0ff77592124b51b580877f513eecd6: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:55:10.243979 9239 mvcc.cc:205] Tried to move back new op lower bound from 7333914665930903552 to 7333914665855377408. Current Snapshot: MvccSnapshot[applied={T|T < 7333914665930903552}]
I20260927 10:55:10.244370 9234 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.244477 9234 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P c9e860429f2f4a95adb75369bac242ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:55:10.245154 9230 tablet_replica.cc:439] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } } }
I20260927 10:55:10.245247 9230 tablet_replica.cc:442] T 51f8337ba92d475fb2fce25c220f546e P 5a0a16b4348244fcb02f48add05a6d8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:55:10.250530 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.246979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.1.46.195:44207)
0927 10:55:10.247151 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.247154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.247164 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:55:10.247213 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.247215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.247253 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.247354 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:10.247609 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.248244 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.249085 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.249431 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.249533 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.249754 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.250199 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.250302 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.250356 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.250404 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 10:55:10.261471 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.257391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:33285 (local address 127.1.46.193:39777)
0927 10:55:10.257512 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.257526 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.257653 (+ 127us) server_negotiation.cc:407] Connection header received
0927 10:55:10.257792 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.257795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.257838 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.257908 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:10.258174 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.258767 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.260491 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.260730 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.260821 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.261103 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.261165 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.261226 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.261336 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.261377 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:10.261526 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.257275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39777 (local address 127.1.46.195:33285)
0927 10:55:10.257567 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:10.257581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:10.257683 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:10.257934 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:10.257937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:10.257947 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:55:10.258073 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.258079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.258868 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.258872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.260344 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.260354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.260840 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.260844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.260936 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.260981 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.261246 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.261252 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:55:10.261316 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 10:55:10.261384 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260927 10:55:10.268116 8872 raft_consensus.cc:1261] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Refusing update from remote peer 5a0a16b4348244fcb02f48add05a6d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.268146 9012 raft_consensus.cc:1261] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Refusing update from remote peer 5a0a16b4348244fcb02f48add05a6d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.268523 9256 consensus_queue.cc:1035] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.268695 9230 consensus_queue.cc:1035] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.272562 9241 mvcc.cc:205] Tried to move back new op lower bound from 7333914666054111232 to 7333914664886857728. Current Snapshot: MvccSnapshot[applied={T|T < 7333914666054111232}]
I20260927 10:55:10.299084 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.291508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:59101 (local address 127.1.46.195:44207)
0927 10:55:10.291611 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.291616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.294747 (+ 3131us) server_negotiation.cc:407] Connection header received
0927 10:55:10.295175 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.295178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.295227 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.295272 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:10.295668 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.296462 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.297809 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.298145 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.298212 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.298603 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.298696 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.298856 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.298889 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.298959 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:55:10.299101 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.291427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:44207 (local address 127.1.46.193:59101)
0927 10:55:10.294652 (+ 3225us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:10.294669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:10.295043 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:10.295401 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:10.295404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:10.295411 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:55:10.295534 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.295540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.296562 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.296565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.297687 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:10.297697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.298311 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.298314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:10.298407 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.298453 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.298788 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.298794 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:55:10.298852 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 10:55:10.298930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3075,"thread_start_us":90,"threads_started":1}
I20260927 10:55:10.327328 8714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:10.327498 8714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:10.329720 8714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Sending DeleteTablet for 3 replicas of tablet 2c11c593214c4524acce6bfba5df6596
I20260927 10:55:10.330298 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c11c593214c4524acce6bfba5df6596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:59314
I20260927 10:55:10.330415 8852 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c11c593214c4524acce6bfba5df6596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44156
I20260927 10:55:10.330490 9250 tablet_replica.cc:330] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: stopping tablet replica
I20260927 10:55:10.330623 9250 raft_consensus.cc:2229] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:10.330732 9251 tablet_replica.cc:330] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: stopping tablet replica
I20260927 10:55:10.330740 9250 raft_consensus.cc:2258] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.330842 9251 raft_consensus.cc:2229] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:10.330950 9251 raft_consensus.cc:2258] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.331615 9250 ts_tablet_manager.cc:1905] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.334501 9250 ts_tablet_manager.cc:1918] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:55:10.334610 9250 log.cc:1196] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/2c11c593214c4524acce6bfba5df6596
I20260927 10:55:10.334897 9250 ts_tablet_manager.cc:1939] T 2c11c593214c4524acce6bfba5df6596 P c9e860429f2f4a95adb75369bac242ae: Deleting consensus metadata
I20260927 10:55:10.335031 9251 ts_tablet_manager.cc:1905] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.335552 8699 catalog_manager.cc:4841] TS c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739): tablet 2c11c593214c4524acce6bfba5df6596 (table test-table [id=4f7ecb62eec147e2a82cf670175c1933]) successfully deleted
I20260927 10:55:10.337620 9251 ts_tablet_manager.cc:1918] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:55:10.337718 9251 log.cc:1196] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/2c11c593214c4524acce6bfba5df6596
I20260927 10:55:10.337973 9251 ts_tablet_manager.cc:1939] T 2c11c593214c4524acce6bfba5df6596 P ed0ff77592124b51b580877f513eecd6: Deleting consensus metadata
I20260927 10:55:10.338635 8699 catalog_manager.cc:4841] TS ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777): tablet 2c11c593214c4524acce6bfba5df6596 (table test-table [id=4f7ecb62eec147e2a82cf670175c1933]) successfully deleted
I20260927 10:55:10.339120 9132 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c11c593214c4524acce6bfba5df6596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:48112
I20260927 10:55:10.339318 9249 tablet_replica.cc:330] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: stopping tablet replica
I20260927 10:55:10.339438 9249 raft_consensus.cc:2229] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:10.339740 9249 raft_consensus.cc:2258] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.340565 9249 ts_tablet_manager.cc:1905] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.343290 9249 ts_tablet_manager.cc:1918] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:55:10.343369 9249 log.cc:1196] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/2c11c593214c4524acce6bfba5df6596
I20260927 10:55:10.343634 9249 ts_tablet_manager.cc:1939] T 2c11c593214c4524acce6bfba5df6596 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting consensus metadata
I20260927 10:55:10.344309 8701 catalog_manager.cc:4841] TS 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207): tablet 2c11c593214c4524acce6bfba5df6596 (table test-table [id=4f7ecb62eec147e2a82cf670175c1933]) successfully deleted
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[8670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:55:10.378202 8764 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:10.367877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.1.46.254:44715)
0927 10:55:10.368023 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.368028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.370759 (+ 2731us) server_negotiation.cc:407] Connection header received
0927 10:55:10.370971 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.370975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.371028 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.371081 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:10.372074 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.372898 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.374119 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.374468 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.374550 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.378078 (+ 3528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47188: BlockingRecv error: recv got EOF from 127.0.0.1:47188 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[8670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM
I20260927 10:55:10.390089 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.380204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.1.46.254:44715)
0927 10:55:10.380312 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.380316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.380784 (+ 468us) server_negotiation.cc:407] Connection header received
0927 10:55:10.380888 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.380890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.380938 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.381027 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:10.381786 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.382525 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.383691 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.384001 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.384095 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.386739 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:10.386742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:10.386745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:10.386770 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:10.388430 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:10.388884 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:10.388888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:10.388889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:10.388972 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:10.389250 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:10.389253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:10.389255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:10.389445 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:10.389568 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.389816 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.389917 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:55:10.392701 8714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:10.393157 8714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:10.398253 8852 tablet_service.cc:1467] Processing CreateTablet for tablet a1e8b6998f504f49ba1a409ccbc533e2 (DEFAULT_TABLE table=test-table [id=abd65b069b2e40e3a88116048664ceaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:10.398383 8992 tablet_service.cc:1467] Processing CreateTablet for tablet a1e8b6998f504f49ba1a409ccbc533e2 (DEFAULT_TABLE table=test-table [id=abd65b069b2e40e3a88116048664ceaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:10.398825 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e8b6998f504f49ba1a409ccbc533e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.399148 9132 tablet_service.cc:1467] Processing CreateTablet for tablet a1e8b6998f504f49ba1a409ccbc533e2 (DEFAULT_TABLE table=test-table [id=abd65b069b2e40e3a88116048664ceaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:10.399401 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e8b6998f504f49ba1a409ccbc533e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.401185 9231 tablet_bootstrap.cc:492] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: Bootstrap starting.
I20260927 10:55:10.401589 9226 tablet_bootstrap.cc:492] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: Bootstrap starting.
I20260927 10:55:10.401970 9231 tablet_bootstrap.cc:654] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.402436 9226 tablet_bootstrap.cc:654] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.403136 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e8b6998f504f49ba1a409ccbc533e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.405380 9227 tablet_bootstrap.cc:492] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: Bootstrap starting.
I20260927 10:55:10.406133 9227 tablet_bootstrap.cc:654] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.407208 9227 tablet_bootstrap.cc:492] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: No bootstrap required, opened a new log
I20260927 10:55:10.407289 9227 ts_tablet_manager.cc:1397] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:10.407653 9227 raft_consensus.cc:348] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.407761 9227 raft_consensus.cc:374] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.407800 9227 raft_consensus.cc:729] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0ff77592124b51b580877f513eecd6, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.407930 9227 consensus_queue.cc:260] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.408159 9227 ts_tablet_manager.cc:1428] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.410072 9226 tablet_bootstrap.cc:492] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: No bootstrap required, opened a new log
I20260927 10:55:10.410156 9226 ts_tablet_manager.cc:1397] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260927 10:55:10.410538 9226 raft_consensus.cc:348] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.410676 9226 raft_consensus.cc:374] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.410705 9226 raft_consensus.cc:729] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0a16b4348244fcb02f48add05a6d8a, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.410813 9226 consensus_queue.cc:260] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.411026 9226 ts_tablet_manager.cc:1428] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.411345 9231 tablet_bootstrap.cc:492] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: No bootstrap required, opened a new log
I20260927 10:55:10.411448 9231 ts_tablet_manager.cc:1397] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260927 10:55:10.411882 9231 raft_consensus.cc:348] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.412000 9231 raft_consensus.cc:374] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.412041 9231 raft_consensus.cc:729] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e860429f2f4a95adb75369bac242ae, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.412179 9231 consensus_queue.cc:260] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.412443 9231 ts_tablet_manager.cc:1428] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.476878 9276 raft_consensus.cc:482] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:10.477032 9276 raft_consensus.cc:504] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.477548 9276 leader_election.cc:288] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739), 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207)
I20260927 10:55:10.477941 9136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e8b6998f504f49ba1a409ccbc533e2" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0a16b4348244fcb02f48add05a6d8a" is_pre_election: true
I20260927 10:55:10.477955 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e8b6998f504f49ba1a409ccbc533e2" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae" is_pre_election: true
I20260927 10:55:10.478098 9136 raft_consensus.cc:2454] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 0.
I20260927 10:55:10.478121 9012 raft_consensus.cc:2454] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 0.
I20260927 10:55:10.478389 8805 leader_election.cc:302] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:10.478588 9276 raft_consensus.cc:2781] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:10.478672 9276 raft_consensus.cc:482] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:10.478724 9276 raft_consensus.cc:3037] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.480077 9276 raft_consensus.cc:504] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.480472 9276 leader_election.cc:288] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 election: Requested vote from peers c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739), 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207)
I20260927 10:55:10.480726 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e8b6998f504f49ba1a409ccbc533e2" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae"
I20260927 10:55:10.480782 9136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e8b6998f504f49ba1a409ccbc533e2" candidate_uuid: "ed0ff77592124b51b580877f513eecd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0a16b4348244fcb02f48add05a6d8a"
I20260927 10:55:10.480839 9012 raft_consensus.cc:3037] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.480870 9136 raft_consensus.cc:3037] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.481868 9136 raft_consensus.cc:2454] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 1.
I20260927 10:55:10.481894 9012 raft_consensus.cc:2454] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0ff77592124b51b580877f513eecd6 in term 1.
I20260927 10:55:10.482147 8805 leader_election.cc:302] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:10.482461 9276 raft_consensus.cc:2781] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:10.482555 9276 raft_consensus.cc:686] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 1 LEADER]: Becoming Leader. State: Replica: ed0ff77592124b51b580877f513eecd6, State: Running, Role: LEADER
I20260927 10:55:10.482728 9276 consensus_queue.cc:237] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.484469 8714 catalog_manager.cc:5495] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed0ff77592124b51b580877f513eecd6 (127.1.46.193). New cstate: current_term: 1 leader_uuid: "ed0ff77592124b51b580877f513eecd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } health_report { overall_health: UNKNOWN } } }
I20260927 10:55:10.506918 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.503047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.1.46.193:39777)
0927 10:55:10.503171 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.503175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.503188 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:10.503232 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.503236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.503292 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.503375 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:10.503633 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.504541 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.505776 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.506121 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.506210 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.506465 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.506538 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.506641 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.506749 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.506793 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:10.509408 9012 raft_consensus.cc:1261] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Refusing update from remote peer ed0ff77592124b51b580877f513eecd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.509409 9136 raft_consensus.cc:1261] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Refusing update from remote peer ed0ff77592124b51b580877f513eecd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.509837 9276 consensus_queue.cc:1035] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.509976 9269 consensus_queue.cc:1035] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.522157 8714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:10.522308 8714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:10.524147 8714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Sending DeleteTablet for 3 replicas of tablet a1e8b6998f504f49ba1a409ccbc533e2
I20260927 10:55:10.524631 8852 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e8b6998f504f49ba1a409ccbc533e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44156
I20260927 10:55:10.524688 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e8b6998f504f49ba1a409ccbc533e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:59314
I20260927 10:55:10.524793 9251 tablet_replica.cc:330] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: stopping tablet replica
I20260927 10:55:10.524842 9250 tablet_replica.cc:330] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: stopping tablet replica
I20260927 10:55:10.524888 9251 raft_consensus.cc:2229] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:10.524941 9250 raft_consensus.cc:2229] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:10.525055 9250 raft_consensus.cc:2258] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.525157 9251 raft_consensus.cc:2258] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.525220 8714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:10.525605 8714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:10.525660 9132 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e8b6998f504f49ba1a409ccbc533e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:48112
I20260927 10:55:10.525825 9249 tablet_replica.cc:330] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: stopping tablet replica
I20260927 10:55:10.525854 9250 ts_tablet_manager.cc:1905] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.525919 9249 raft_consensus.cc:2229] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:10.526014 9249 raft_consensus.cc:2258] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.526049 9251 ts_tablet_manager.cc:1905] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.528309 9250 ts_tablet_manager.cc:1918] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:10.528358 9251 ts_tablet_manager.cc:1918] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:10.528383 9250 log.cc:1196] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/a1e8b6998f504f49ba1a409ccbc533e2
I20260927 10:55:10.528424 9251 log.cc:1196] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/a1e8b6998f504f49ba1a409ccbc533e2
I20260927 10:55:10.528630 9250 ts_tablet_manager.cc:1939] T a1e8b6998f504f49ba1a409ccbc533e2 P c9e860429f2f4a95adb75369bac242ae: Deleting consensus metadata
I20260927 10:55:10.528676 9251 ts_tablet_manager.cc:1939] T a1e8b6998f504f49ba1a409ccbc533e2 P ed0ff77592124b51b580877f513eecd6: Deleting consensus metadata
I20260927 10:55:10.528714 9249 ts_tablet_manager.cc:1905] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.529311 8699 catalog_manager.cc:4841] TS c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739): tablet a1e8b6998f504f49ba1a409ccbc533e2 (table test-table [id=abd65b069b2e40e3a88116048664ceaa]) successfully deleted
I20260927 10:55:10.529531 8699 catalog_manager.cc:4841] TS ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777): tablet a1e8b6998f504f49ba1a409ccbc533e2 (table test-table [id=abd65b069b2e40e3a88116048664ceaa]) successfully deleted
I20260927 10:55:10.529874 8852 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8b8ebb85894104adb68a6dc8150868 (DEFAULT_TABLE table=test-table [id=b513a8f97d234dbabd7da1bc1a71c95a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:10.529965 9132 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8b8ebb85894104adb68a6dc8150868 (DEFAULT_TABLE table=test-table [id=b513a8f97d234dbabd7da1bc1a71c95a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:10.529979 8992 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8b8ebb85894104adb68a6dc8150868 (DEFAULT_TABLE table=test-table [id=b513a8f97d234dbabd7da1bc1a71c95a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:10.530083 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8b8ebb85894104adb68a6dc8150868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.530159 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8b8ebb85894104adb68a6dc8150868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.530198 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8b8ebb85894104adb68a6dc8150868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:10.531008 9249 ts_tablet_manager.cc:1918] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:10.531090 9249 log.cc:1196] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/a1e8b6998f504f49ba1a409ccbc533e2
I20260927 10:55:10.531332 9249 ts_tablet_manager.cc:1939] T a1e8b6998f504f49ba1a409ccbc533e2 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting consensus metadata
I20260927 10:55:10.531919 8701 catalog_manager.cc:4841] TS 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207): tablet a1e8b6998f504f49ba1a409ccbc533e2 (table test-table [id=abd65b069b2e40e3a88116048664ceaa]) successfully deleted
I20260927 10:55:10.532155 9227 tablet_bootstrap.cc:492] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6: Bootstrap starting.
I20260927 10:55:10.532213 9226 tablet_bootstrap.cc:492] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: Bootstrap starting.
I20260927 10:55:10.532373 9231 tablet_bootstrap.cc:492] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: Bootstrap starting.
I20260927 10:55:10.532884 9227 tablet_bootstrap.cc:654] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.533022 9226 tablet_bootstrap.cc:654] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.533035 9231 tablet_bootstrap.cc:654] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:10.533986 9226 tablet_bootstrap.cc:492] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: No bootstrap required, opened a new log
I20260927 10:55:10.534071 9226 ts_tablet_manager.cc:1397] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:10.534332 9227 tablet_bootstrap.cc:492] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6: No bootstrap required, opened a new log
I20260927 10:55:10.534430 9227 ts_tablet_manager.cc:1397] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:10.534431 9226 raft_consensus.cc:348] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.534538 9226 raft_consensus.cc:374] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.534602 9226 raft_consensus.cc:729] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0a16b4348244fcb02f48add05a6d8a, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.534731 9226 consensus_queue.cc:260] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.534866 9227 raft_consensus.cc:348] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.534953 9226 ts_tablet_manager.cc:1428] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.534978 9227 raft_consensus.cc:374] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.535022 9227 raft_consensus.cc:729] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0ff77592124b51b580877f513eecd6, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.535137 9227 consensus_queue.cc:260] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.535377 9227 ts_tablet_manager.cc:1428] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.536007 9231 tablet_bootstrap.cc:492] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: No bootstrap required, opened a new log
I20260927 10:55:10.536096 9231 ts_tablet_manager.cc:1397] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 10:55:10.536422 9231 raft_consensus.cc:348] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.536525 9231 raft_consensus.cc:374] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:10.536559 9231 raft_consensus.cc:729] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e860429f2f4a95adb75369bac242ae, State: Initialized, Role: FOLLOWER
I20260927 10:55:10.536674 9231 consensus_queue.cc:260] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.536893 9231 ts_tablet_manager.cc:1428] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:55:10.675474 9230 raft_consensus.cc:482] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:10.675617 9230 raft_consensus.cc:504] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.676007 9230 leader_election.cc:288] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777), c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739)
I20260927 10:55:10.676530 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b8ebb85894104adb68a6dc8150868" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0ff77592124b51b580877f513eecd6" is_pre_election: true
I20260927 10:55:10.676553 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b8ebb85894104adb68a6dc8150868" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae" is_pre_election: true
I20260927 10:55:10.676687 8872 raft_consensus.cc:2454] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 0.
I20260927 10:55:10.676694 9012 raft_consensus.cc:2454] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 0.
I20260927 10:55:10.676947 9083 leader_election.cc:302] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:10.677103 9230 raft_consensus.cc:2781] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:10.677171 9230 raft_consensus.cc:482] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:10.677235 9230 raft_consensus.cc:3037] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.678174 9230 raft_consensus.cc:504] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.678522 9230 leader_election.cc:288] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 election: Requested vote from peers ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777), c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739)
I20260927 10:55:10.678867 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b8ebb85894104adb68a6dc8150868" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0ff77592124b51b580877f513eecd6"
I20260927 10:55:10.678941 9012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b8ebb85894104adb68a6dc8150868" candidate_uuid: "5a0a16b4348244fcb02f48add05a6d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e860429f2f4a95adb75369bac242ae"
I20260927 10:55:10.679029 9012 raft_consensus.cc:3037] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.679016 8872 raft_consensus.cc:3037] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:10.680330 9012 raft_consensus.cc:2454] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 1.
I20260927 10:55:10.680330 8872 raft_consensus.cc:2454] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0a16b4348244fcb02f48add05a6d8a in term 1.
I20260927 10:55:10.680635 9083 leader_election.cc:302] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0a16b4348244fcb02f48add05a6d8a, ed0ff77592124b51b580877f513eecd6; no voters:
I20260927 10:55:10.680810 9230 raft_consensus.cc:2781] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:10.680926 9230 raft_consensus.cc:686] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 LEADER]: Becoming Leader. State: Replica: 5a0a16b4348244fcb02f48add05a6d8a, State: Running, Role: LEADER
I20260927 10:55:10.681149 9230 consensus_queue.cc:237] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } }
I20260927 10:55:10.682821 8714 catalog_manager.cc:5495] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195). New cstate: current_term: 1 leader_uuid: "5a0a16b4348244fcb02f48add05a6d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0a16b4348244fcb02f48add05a6d8a" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 44207 } health_report { overall_health: HEALTHY } } }
I20260927 10:55:10.697036 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.694170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.1.46.195:44207)
0927 10:55:10.694267 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:10.694278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:10.694312 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:55:10.694413 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:10.694415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:10.694449 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:10.694506 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:10.694747 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.695299 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.696020 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:10.696290 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:10.696393 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:10.696641 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:10.696705 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:10.696780 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:10.696868 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 10:55:10.696911 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 10:55:10.702893 9012 raft_consensus.cc:1261] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Refusing update from remote peer 5a0a16b4348244fcb02f48add05a6d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.703001 8872 raft_consensus.cc:1261] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Refusing update from remote peer 5a0a16b4348244fcb02f48add05a6d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:10.703209 9230 consensus_queue.cc:1035] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e860429f2f4a95adb75369bac242ae" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.703356 9283 consensus_queue.cc:1035] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0ff77592124b51b580877f513eecd6" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:10.732795 8712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:10.732942 8712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:10.734644 8712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47e20a38834e4d488c644002c9b2320d: Sending DeleteTablet for 3 replicas of tablet 2b8b8ebb85894104adb68a6dc8150868
I20260927 10:55:10.735210 8852 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8b8ebb85894104adb68a6dc8150868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:44156
I20260927 10:55:10.735220 9132 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8b8ebb85894104adb68a6dc8150868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:48112
I20260927 10:55:10.735267 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8b8ebb85894104adb68a6dc8150868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:10 UTC) from {username='oryx', principal='oryx/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:59314
I20260927 10:55:10.735374 9251 tablet_replica.cc:330] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6: stopping tablet replica
I20260927 10:55:10.735380 9249 tablet_replica.cc:330] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: stopping tablet replica
I20260927 10:55:10.735417 9250 tablet_replica.cc:330] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: stopping tablet replica
I20260927 10:55:10.735463 9251 raft_consensus.cc:2229] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:10.735481 9249 raft_consensus.cc:2229] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:10.735510 9250 raft_consensus.cc:2229] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:10.735538 9251 raft_consensus.cc:2258] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.735594 9250 raft_consensus.cc:2258] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.735636 9249 raft_consensus.cc:2258] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:10.736258 9249 ts_tablet_manager.cc:1905] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.736274 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 8787
I20260927 10:55:10.736312 9250 ts_tablet_manager.cc:1905] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.736419 9251 ts_tablet_manager.cc:1905] T 2b8b8ebb85894104adb68a6dc8150868 P ed0ff77592124b51b580877f513eecd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:10.738699 9250 ts_tablet_manager.cc:1918] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:55:10.738783 9250 log.cc:1196] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/2b8b8ebb85894104adb68a6dc8150868
I20260927 10:55:10.739061 9250 ts_tablet_manager.cc:1939] T 2b8b8ebb85894104adb68a6dc8150868 P c9e860429f2f4a95adb75369bac242ae: Deleting consensus metadata
I20260927 10:55:10.739248 9249 ts_tablet_manager.cc:1918] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:55:10.739341 9249 log.cc:1196] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/2b8b8ebb85894104adb68a6dc8150868
I20260927 10:55:10.739615 9249 ts_tablet_manager.cc:1939] T 2b8b8ebb85894104adb68a6dc8150868 P 5a0a16b4348244fcb02f48add05a6d8a: Deleting consensus metadata
I20260927 10:55:10.739768 8699 catalog_manager.cc:4841] TS c9e860429f2f4a95adb75369bac242ae (127.1.46.194:39739): tablet 2b8b8ebb85894104adb68a6dc8150868 (table test-table [id=b513a8f97d234dbabd7da1bc1a71c95a]) successfully deleted
I20260927 10:55:10.740254 8701 catalog_manager.cc:4841] TS 5a0a16b4348244fcb02f48add05a6d8a (127.1.46.195:44207): tablet 2b8b8ebb85894104adb68a6dc8150868 (table test-table [id=b513a8f97d234dbabd7da1bc1a71c95a]) successfully deleted
W20260927 10:55:10.745281 8699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:39777 (error 108)
I20260927 10:55:10.745496 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 8927
I20260927 10:55:10.746682 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:10.746230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39777 (local address 127.0.0.1:44176)
0927 10:55:10.746434 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:10.746516 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260927 10:55:10.746884 8699 catalog_manager.cc:4584] TS ed0ff77592124b51b580877f513eecd6 (127.1.46.193:39777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b8b8ebb85894104adb68a6dc8150868: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39777: connect: Connection refused (error 111)
I20260927 10:55:10.752302 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 9067
I20260927 10:55:10.758613 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 8683
2026-09-27T10:55:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3319 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[9319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[9319](info): set up 2 sockets
Sep 27 10:55:10 dist-test-slave-cffh krb5kdc[9319](info): commencing operation
krb5kdc: starting...
W20260927 10:55:11.795234 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:55:11 dist-test-slave-cffh krb5kdc[9319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:55:11.910012 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:33057
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:12.015151 9332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:12.015406 9332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:12.015475 9332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:12.019412 9332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:12.019484 9332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:12.019522 9332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:12.019577 9332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:12.019615 9332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:12.024675 9332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:33057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:12.025611 9332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:12.026643 9332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:12.034550 9338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:12.034545 9340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:12.034652 9332 server_base.cc:1034] running on GCE node
W20260927 10:55:12.034545 9337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:12.035467 9332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:12.036480 9332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:12.037633 9332 hybrid_clock.cc:648] HybridClock initialized: now 1790506512037607 us; error 47 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T10:55:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1287 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:12 dist-test-slave-cffh krb5kdc[9345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:12 dist-test-slave-cffh krb5kdc[9345](info): set up 2 sockets
Sep 27 10:55:12 dist-test-slave-cffh krb5kdc[9345](info): commencing operation
krb5kdc: starting...
W20260927 10:55:13.075150 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:55:13 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:55:13.192076 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:41257
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:41257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:13.297276 9358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:13.297573 9358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:13.297648 9358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:13.301592 9358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:13.301676 9358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:13.301730 9358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:13.301785 9358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:13.301822 9358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:13.307181 9358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:41257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:41257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:13.308197 9358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:13.309199 9358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:13.317354 9364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:13.317344 9363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:13.317344 9366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:13.317559 9358 server_base.cc:1034] running on GCE node
I20260927 10:55:13.318126 9358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:13.319204 9358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:13.320389 9358 hybrid_clock.cc:648] HybridClock initialized: now 1790506513320374 us; error 25 us; skew 500 ppm
Sep 27 10:55:13 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:13.328163 9358 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:55:13.329015 9358 webserver.cc:460] Webserver started at http://127.1.46.254:44181/ using document root <none> and password file <none>
I20260927 10:55:13.329531 9358 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:13.329613 9358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:13.329828 9358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:13.332043 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "382856575ee14efba9b1ab048e1ccd93"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "a635c89bc097d342dd8005be788001e7"
server_key_iv: "954fd58fbd1fbb869e550e1a6f132c44"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.332609 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "382856575ee14efba9b1ab048e1ccd93"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "a635c89bc097d342dd8005be788001e7"
server_key_iv: "954fd58fbd1fbb869e550e1a6f132c44"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.336113 9358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:55:13.338387 9373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:13.339560 9358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:13.339710 9358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "382856575ee14efba9b1ab048e1ccd93"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "a635c89bc097d342dd8005be788001e7"
server_key_iv: "954fd58fbd1fbb869e550e1a6f132c44"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.339828 9358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:13.360574 9358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:13.363332 9358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:13.363516 9358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:13.370787 9358 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:41257
I20260927 10:55:13.370812 9425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:41257 every 8 connection(s)
I20260927 10:55:13.371645 9358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:13.374286 9426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:13.377844 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 9358
I20260927 10:55:13.377947 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:55:13.378224 1211 external_mini_cluster.cc:1428] Setting key 8c1fe2b1eabdf968f7aa2f9452aa2bcd
I20260927 10:55:13.380184 9426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Bootstrap starting.
I20260927 10:55:13.382959 9426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:13.383832 9426 log.cc:824] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:13.386270 9426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: No bootstrap required, opened a new log
Sep 27 10:55:13 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:13.390502 9426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:13.390820 9426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:13.390901 9426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382856575ee14efba9b1ab048e1ccd93, State: Initialized, Role: FOLLOWER
I20260927 10:55:13.391314 9426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:13.391454 9426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:13.391523 9426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:13.391634 9426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:13.392429 9426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:13.392755 9426 leader_election.cc:302] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382856575ee14efba9b1ab048e1ccd93; no voters:
I20260927 10:55:13.393071 9426 leader_election.cc:288] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:13.393134 9431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:13.393384 9431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 LEADER]: Becoming Leader. State: Replica: 382856575ee14efba9b1ab048e1ccd93, State: Running, Role: LEADER
I20260927 10:55:13.393710 9431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:13.394331 9426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:13.394415 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:13.379572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.1.46.254:41257)
0927 10:55:13.380011 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:13.380030 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:13.380074 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:55:13.380971 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:13.380993 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:13.381440 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:13.381802 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:13.382427 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.383983 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.385330 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.386101 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.386266 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:13.389480 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:13.389486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:13.389509 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:13.389534 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:13.391514 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:13.391943 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:13.391949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:13.391956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:13.392074 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:13.392366 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:13.392370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:13.392373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:13.392804 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:13.392999 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:13.393317 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 10:55:13.393622 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260927 10:55:13.394848 9431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382856575ee14efba9b1ab048e1ccd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } } }
I20260927 10:55:13.394969 9431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:13.395159 9431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382856575ee14efba9b1ab048e1ccd93. Latest consensus state: current_term: 1 leader_uuid: "382856575ee14efba9b1ab048e1ccd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } } }
I20260927 10:55:13.395252 9431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:13.395452 9436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:13.398808 9436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:13.404723 9436 catalog_manager.cc:1347] Generated new cluster ID: 4995008afc4d4f85ac7e2d14e1fe05b4
I20260927 10:55:13.404783 9436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:13.413185 9436 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:13.414011 9436 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:13.418093 9436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Generated new TSK 0
I20260927 10:55:13.418735 9436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:55:13.449628 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41257
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:13.557226 9452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:13.557475 9452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:13.557552 9452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:13.561719 9452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:13.561807 9452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:13.561898 9452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:13.567111 9452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:13.568023 9452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:13.568948 9452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:13.577970 9460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:13.577976 9458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:13.578020 9457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:13.578301 9452 server_base.cc:1034] running on GCE node
I20260927 10:55:13.578811 9452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:13.579398 9452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:13.580598 9452 hybrid_clock.cc:648] HybridClock initialized: now 1790506513580552 us; error 68 us; skew 500 ppm
Sep 27 10:55:13 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:13.583251 9452 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:55:13.583984 9452 webserver.cc:460] Webserver started at http://127.1.46.193:37639/ using document root <none> and password file <none>
I20260927 10:55:13.584501 9452 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:13.584556 9452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:13.584707 9452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:13.586547 9452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "853655c5efe7480a87b6961e936e0afe"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "ea927d0466bb4119eb8875c24a8b4fcc"
server_key_iv: "ba06bc4b8d6929e2951574d2651a6935"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.587095 9452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "853655c5efe7480a87b6961e936e0afe"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "ea927d0466bb4119eb8875c24a8b4fcc"
server_key_iv: "ba06bc4b8d6929e2951574d2651a6935"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.590461 9452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:13.592716 9467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:13.593822 9452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:13.594020 9452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "853655c5efe7480a87b6961e936e0afe"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "ea927d0466bb4119eb8875c24a8b4fcc"
server_key_iv: "ba06bc4b8d6929e2951574d2651a6935"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.594103 9452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:13.613145 9452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:13.616250 9452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:13.616479 9452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:13.617044 9452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:13.617834 9452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:13.617906 9452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:13.617975 9452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:13.618028 9452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:13.626502 9452 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:35447
I20260927 10:55:13.626585 9580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:35447 every 8 connection(s)
I20260927 10:55:13.627384 9452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:13.635144 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 9452
I20260927 10:55:13.635291 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:13.635591 1211 external_mini_cluster.cc:1428] Setting key c0b8572e4c916b33c1a25fe860a165e6
Sep 27 10:55:13 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:13.641072 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:13.629055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:35505 (local address 127.1.46.254:41257)
0927 10:55:13.629194 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:13.629198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:13.629939 (+ 741us) server_negotiation.cc:407] Connection header received
0927 10:55:13.631318 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:13.631321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:13.631375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:13.631481 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:13.632554 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.633113 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.634235 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.634464 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.634575 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:13.637283 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:13.637287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:13.637290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:13.637318 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:13.639014 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:13.639612 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:13.639614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:13.639615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:13.639684 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:13.640054 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:13.640056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:13.640057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:13.640216 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:13.640293 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:13.640869 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 10:55:13.640960 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:13.642241 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:13.629398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:35505)
0927 10:55:13.629803 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:13.629842 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:13.631110 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:13.631642 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:13.631649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:13.631969 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:13.632412 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:13.632426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.633244 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.633248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:13.634097 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:13.634105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.634647 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.634652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:13.634808 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:13.635431 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:13.635447 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:13.637123 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:13.639120 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:13.639127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:13.639142 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:13.639516 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:13.639778 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:13.639782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:13.639786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:13.639961 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:13.640324 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:13.640333 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:13.640624 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 10:55:13.640839 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 10:55:13.641168 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
I20260927 10:55:13.643522 9581 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41257
I20260927 10:55:13.643800 9581 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:13.644302 9581 heartbeater.cc:511] Master 127.1.46.254:41257 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:55:13.645771 9390 ts_manager.cc:194] Registered new tserver with Master: 853655c5efe7480a87b6961e936e0afe (127.1.46.193:35447)
I20260927 10:55:13.647042 9390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:35505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:55:13.661389 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41257
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:13.766588 9586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:13.766814 9586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:13.766853 9586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:13.770682 9586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:13.770779 9586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:13.770874 9586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:13.776026 9586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:13.776929 9586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:13.777848 9586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:13.786600 9591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:13.786616 9594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:13.786640 9592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:13.786784 9586 server_base.cc:1034] running on GCE node
I20260927 10:55:13.787243 9586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:13.787803 9586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:13.788987 9586 hybrid_clock.cc:648] HybridClock initialized: now 1790506513788957 us; error 32 us; skew 500 ppm
Sep 27 10:55:13 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:13.791575 9586 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:55:13.792371 9586 webserver.cc:460] Webserver started at http://127.1.46.194:46577/ using document root <none> and password file <none>
I20260927 10:55:13.792929 9586 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:13.793016 9586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:13.793241 9586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:13.795356 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "173d0f6693a94a4a93a2c10723d23d94"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "2e1a4eb5740ff52231592c24252774b4"
server_key_iv: "42af68bac61d78bcdad03107f216a229"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.795989 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "173d0f6693a94a4a93a2c10723d23d94"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "2e1a4eb5740ff52231592c24252774b4"
server_key_iv: "42af68bac61d78bcdad03107f216a229"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.799616 9586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 10:55:13.801860 9601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:13.803076 9586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:13.803206 9586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "173d0f6693a94a4a93a2c10723d23d94"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "2e1a4eb5740ff52231592c24252774b4"
server_key_iv: "42af68bac61d78bcdad03107f216a229"
server_key_version: "encryptionkey@0"
I20260927 10:55:13.803318 9586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:13.825384 9586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:13.828202 9586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:13.828428 9586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:13.828979 9586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:13.829754 9586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:13.829823 9586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:13.829890 9586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:13.829936 9586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:13.838343 9586 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:44445
I20260927 10:55:13.838366 9714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:44445 every 8 connection(s)
I20260927 10:55:13.839215 9586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:13.846921 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 9586
I20260927 10:55:13.847060 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:13.847422 1211 external_mini_cluster.cc:1428] Setting key 0430649f5e25df081b73060e0f0d5e9e
Sep 27 10:55:13 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:13.852856 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:13.841000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:51431 (local address 127.1.46.254:41257)
0927 10:55:13.841121 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:13.841124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:13.841878 (+ 754us) server_negotiation.cc:407] Connection header received
0927 10:55:13.843395 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:13.843398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:13.843462 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:13.843584 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:13.844743 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.845278 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.846396 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.846694 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.846800 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:13.849189 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:13.849194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:13.849197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:13.849230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:13.850824 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:13.851298 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:13.851301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:13.851303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:13.851392 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:13.851669 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:13.851675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:13.851679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:13.851930 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:13.852028 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:13.852477 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 10:55:13.852607 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:55:13.853497 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:13.841306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:51431)
0927 10:55:13.841726 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:13.841772 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:13.843167 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:13.843755 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:13.843761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:13.844084 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:13.844603 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:13.844616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.845397 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.845401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:13.846252 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:13.846260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:13.846831 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:13.846835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:13.846992 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:13.847616 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:13.847630 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:13.849012 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:13.850929 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:13.850933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:13.850943 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:13.851198 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:13.851480 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:13.851482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:13.851483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:13.851598 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:13.852025 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:13.852029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:13.852234 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 10:55:13.852397 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:55:13.852617 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260927 10:55:13.854746 9715 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41257
I20260927 10:55:13.854962 9715 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:13.855381 9715 heartbeater.cc:511] Master 127.1.46.254:41257 requested a full tablet report, sending...
I20260927 10:55:13.856489 9390 ts_manager.cc:194] Registered new tserver with Master: 173d0f6693a94a4a93a2c10723d23d94 (127.1.46.194:44445)
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:55:13.857265 9390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:51431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:55:13.874007 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41257
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:13.985998 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:13.986232 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:13.986276 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:13.990281 9720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:13.990337 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:13.990424 9720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:13.995813 9720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:13.996807 9720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:13.997799 9720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:14.007157 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.007145 9726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.007145 9725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:14.007402 9720 server_base.cc:1034] running on GCE node
I20260927 10:55:14.007894 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:14.008477 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:14.009660 9720 hybrid_clock.cc:648] HybridClock initialized: now 1790506514009649 us; error 30 us; skew 500 ppm
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506514, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:14.012274 9720 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:55:14.013037 9720 webserver.cc:460] Webserver started at http://127.1.46.195:45307/ using document root <none> and password file <none>
I20260927 10:55:14.013564 9720 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:14.013649 9720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:14.013886 9720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:14.015867 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "c1914f9868b74c1997e4f4540fc7415f"
format_stamp: "Formatted at 2026-09-27 10:55:14 on dist-test-slave-cffh"
server_key: "94a077dbf0ed5f005d1420b06756602e"
server_key_iv: "6ca73c89caa128c68851703ee6d6a5b1"
server_key_version: "encryptionkey@0"
I20260927 10:55:14.016444 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "c1914f9868b74c1997e4f4540fc7415f"
format_stamp: "Formatted at 2026-09-27 10:55:14 on dist-test-slave-cffh"
server_key: "94a077dbf0ed5f005d1420b06756602e"
server_key_iv: "6ca73c89caa128c68851703ee6d6a5b1"
server_key_version: "encryptionkey@0"
I20260927 10:55:14.019873 9720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:14.022029 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.023212 9720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:14.023339 9720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "c1914f9868b74c1997e4f4540fc7415f"
format_stamp: "Formatted at 2026-09-27 10:55:14 on dist-test-slave-cffh"
server_key: "94a077dbf0ed5f005d1420b06756602e"
server_key_iv: "6ca73c89caa128c68851703ee6d6a5b1"
server_key_version: "encryptionkey@0"
I20260927 10:55:14.023439 9720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:14.037717 9720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:14.040328 9720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:14.040506 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:14.041050 9720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:14.041790 9720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:14.041857 9720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.041925 9720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:14.041970 9720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.050954 9720 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:34493
I20260927 10:55:14.050992 9848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:34493 every 8 connection(s)
I20260927 10:55:14.051808 9720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:14.059996 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 9720
I20260927 10:55:14.060127 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:14.060410 1211 external_mini_cluster.cc:1428] Setting key be8a5df1dac7752a773e0a9a4d7c4a04
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506514, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:14.064700 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.053485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:54813 (local address 127.1.46.254:41257)
0927 10:55:14.053643 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.053648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.054418 (+ 770us) server_negotiation.cc:407] Connection header received
0927 10:55:14.055691 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.055694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.055738 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.055808 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.056834 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.057371 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.058430 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.058715 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.058792 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.061242 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:14.061245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:14.061247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:14.061268 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:14.062901 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:14.063361 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:14.063364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:14.063365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:14.063437 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:14.063819 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:14.063821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:14.063822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:14.063974 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:14.064045 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:14.064491 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 10:55:14.064579 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:55:14.065399 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.053865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:54813)
0927 10:55:14.054269 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.054306 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.055494 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.055953 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.055959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.056274 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.056717 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.056729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.057484 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.057487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.058333 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.058341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.058828 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.058831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.058958 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.059413 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.059427 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.061090 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:14.063002 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:14.063006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:14.063016 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:14.063265 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:14.063525 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:14.063527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:14.063528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:14.063725 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:14.064067 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:14.064072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:14.064273 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 10:55:14.064429 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 10:55:14.064643 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
I20260927 10:55:14.066506 9849 heartbeater.cc:347] Connected to a master server at 127.1.46.254:41257
I20260927 10:55:14.066780 9849 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:14.067171 9849 heartbeater.cc:511] Master 127.1.46.254:41257 requested a full tablet report, sending...
I20260927 10:55:14.068128 9390 ts_manager.cc:194] Registered new tserver with Master: c1914f9868b74c1997e4f4540fc7415f (127.1.46.195:34493)
I20260927 10:55:14.068642 9390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:54813
I20260927 10:55:14.074038 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab.
Entry for principal oryx/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab.
WARNING: no policy specified for oryx/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab.
Entry for principal oryx/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab.
WARNING: no policy specified for oryx/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab.
Entry for principal oryx/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab.
I20260927 10:55:14.141471 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 9452
I20260927 10:55:14.147307 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 9586
I20260927 10:55:14.152413 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 9720
I20260927 10:55:14.157428 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 9358
I20260927 10:55:14.163516 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:41257
--webserver_interface=127.1.46.254
--webserver_port=44181
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:41257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:14.268669 9858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:14.268931 9858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:14.269023 9858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:14.273025 9858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:14.273109 9858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:14.273146 9858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:14.273216 9858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:14.273276 9858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:14.278329 9858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:41257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:41257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=44181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:14.279322 9858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:14.280267 9858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:14.288400 9863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.288411 9866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:14.288463 9858 server_base.cc:1034] running on GCE node
W20260927 10:55:14.288663 9864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:14.289153 9858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:14.290169 9858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:14.291379 9858 hybrid_clock.cc:648] HybridClock initialized: now 1790506514291346 us; error 51 us; skew 500 ppm
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506514, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:14.294078 9858 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:55:14.294988 9858 webserver.cc:460] Webserver started at http://127.1.46.254:44181/ using document root <none> and password file <none>
I20260927 10:55:14.295480 9858 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:14.295550 9858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:14.299152 9858 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:14.301139 9873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.302374 9858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:14.302507 9858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "382856575ee14efba9b1ab048e1ccd93"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "a635c89bc097d342dd8005be788001e7"
server_key_iv: "954fd58fbd1fbb869e550e1a6f132c44"
server_key_version: "encryptionkey@0"
I20260927 10:55:14.303086 9858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:14.329419 9858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:14.332402 9858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:14.332553 9858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:14.339619 9858 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:41257
I20260927 10:55:14.339639 9925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:41257 every 8 connection(s)
I20260927 10:55:14.340653 9858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:14.344769 9926 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:55:14.346376 9926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Bootstrap starting.
I20260927 10:55:14.349328 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 9858
I20260927 10:55:14.349782 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:35447
--local_ip_for_outbound_sockets=127.1.46.193
--tserver_master_addrs=127.1.46.254:41257
--webserver_port=37639
--webserver_interface=127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:55:14.365639 9926 log.cc:824] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:14.372383 9926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:55:14.372754 9926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Bootstrap complete.
I20260927 10:55:14.377067 9926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:14.377485 9926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382856575ee14efba9b1ab048e1ccd93, State: Initialized, Role: FOLLOWER
I20260927 10:55:14.377993 9926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:14.378150 9926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:14.378244 9926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:14.378365 9926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:55:14.380311 9926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:14.380661 9926 leader_election.cc:302] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382856575ee14efba9b1ab048e1ccd93; no voters:
I20260927 10:55:14.380999 9926 leader_election.cc:288] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:55:14.381088 9930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:55:14.381282 9930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [term 2 LEADER]: Becoming Leader. State: Replica: 382856575ee14efba9b1ab048e1ccd93, State: Running, Role: LEADER
I20260927 10:55:14.381633 9930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } }
I20260927 10:55:14.382264 9926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:14.383168 9931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "382856575ee14efba9b1ab048e1ccd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } } }
I20260927 10:55:14.383306 9931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:14.383576 9932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382856575ee14efba9b1ab048e1ccd93. Latest consensus state: current_term: 2 leader_uuid: "382856575ee14efba9b1ab048e1ccd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382856575ee14efba9b1ab048e1ccd93" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 41257 } } }
I20260927 10:55:14.383643 9932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:14.383754 9939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:14.387039 9939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:14.388028 9939 catalog_manager.cc:1259] Loaded cluster ID: 4995008afc4d4f85ac7e2d14e1fe05b4
I20260927 10:55:14.388131 9939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:14.390522 9939 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:14.391249 9939 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 382856575ee14efba9b1ab048e1ccd93: Loaded TSK: 0
I20260927 10:55:14.392520 9939 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:55:14.469636 9928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:14.469931 9928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:14.470016 9928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:14.474023 9928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:14.474121 9928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:14.474223 9928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:14.479523 9928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.193.keytab
--principal=oryx/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:35447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=37639
--tserver_master_addrs=127.1.46.254:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:14.480611 9928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:14.481642 9928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:14.491802 9953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.491808 9952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.492262 9955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:14.492322 9928 server_base.cc:1034] running on GCE node
I20260927 10:55:14.492712 9928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:14.493413 9928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:14.494638 9928 hybrid_clock.cc:648] HybridClock initialized: now 1790506514494606 us; error 45 us; skew 500 ppm
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506514, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:14.497591 9928 init.cc:377] Logged in from keytab as oryx/127.1.46.193@KRBTEST.COM (short username oryx)
I20260927 10:55:14.498646 9928 webserver.cc:460] Webserver started at http://127.1.46.193:37639/ using document root <none> and password file <none>
I20260927 10:55:14.499224 9928 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:14.499303 9928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:14.503324 9928 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:14.505584 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.507218 9928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:14.507346 9928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "853655c5efe7480a87b6961e936e0afe"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "ea927d0466bb4119eb8875c24a8b4fcc"
server_key_iv: "ba06bc4b8d6929e2951574d2651a6935"
server_key_version: "encryptionkey@0"
I20260927 10:55:14.507999 9928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:14.525815 9928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:14.529558 9928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:14.529784 9928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:14.530469 9928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:14.531533 9928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:14.531599 9928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.531651 9928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:14.531672 9928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.540973 9928 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:35447
I20260927 10:55:14.541062 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:35447 every 8 connection(s)
I20260927 10:55:14.542002 9928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:14.546326 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 9928
I20260927 10:55:14.546883 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:44445
--local_ip_for_outbound_sockets=127.1.46.194
--tserver_master_addrs=127.1.46.254:41257
--webserver_port=46577
--webserver_interface=127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.555349 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.544109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:54309)
0927 10:55:14.544558 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.544609 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.545802 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.547192 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.547201 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.547571 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.548032 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.548047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.549343 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.549348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.550220 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.550230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.551069 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.551074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.551279 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.551986 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.552002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.554630 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":136,"threads_started":1}
W20260927 10:55:14.555809 10076 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:41257 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.46.254:41257: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
W20260927 10:55:14.555857 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.544012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:54309 (local address 127.1.46.254:41257)
0927 10:55:14.544456 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.544471 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.544728 (+ 257us) server_negotiation.cc:407] Connection header received
0927 10:55:14.546208 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.546223 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.546653 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.547012 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.548218 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.549186 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.550359 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.550898 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.551084 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.554912 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:54309: BlockingRecv error: recv got EOF from 127.1.46.193:54309 (error 108)
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.561273 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.556230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:38803)
0927 10:55:14.556370 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.556407 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.556481 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.556719 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.556721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.556880 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.556963 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.556969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.557674 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.557680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.558221 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.558228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.558709 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.558713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.558815 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.559296 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.559308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.561139 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 10:55:14.561354 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.556343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:38803 (local address 127.1.46.254:41257)
0927 10:55:14.556473 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.556477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.556490 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:14.556568 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.556572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.556618 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.556705 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.557076 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.557566 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.558343 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.558554 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.558711 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.561248 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:38803: BlockingRecv error: recv got EOF from 127.1.46.193:38803 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.566807 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.561762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:51853)
0927 10:55:14.561995 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.562009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.562111 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.562346 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.562350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.562490 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.562550 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.562555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.563281 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.563285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.563794 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.563802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.564224 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.564227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.564312 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.564783 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.564794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.566647 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174}
W20260927 10:55:14.566900 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.561885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:51853 (local address 127.1.46.254:41257)
0927 10:55:14.561979 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.561982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.562077 (+ 95us) server_negotiation.cc:407] Connection header received
0927 10:55:14.562194 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.562197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.562240 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.562320 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.562691 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.563167 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.563908 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.564120 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.564225 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.566777 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:51853: BlockingRecv error: recv got EOF from 127.1.46.193:51853 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 10:55:14.567040 10076 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:55:14.661299 10080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:14.661552 10080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:14.661594 10080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:14.665695 10080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:14.665753 10080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:14.665812 10080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:14.671026 10080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.194.keytab
--principal=oryx/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:44445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=46577
--tserver_master_addrs=127.1.46.254:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:14.672037 10080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:14.673044 10080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:14.682382 10086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.682403 10088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.682428 10085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:14.682621 10080 server_base.cc:1034] running on GCE node
I20260927 10:55:14.683314 10080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:14.683990 10080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:14.685179 10080 hybrid_clock.cc:648] HybridClock initialized: now 1790506514685167 us; error 68 us; skew 500 ppm
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506514, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:14.688059 10080 init.cc:377] Logged in from keytab as oryx/127.1.46.194@KRBTEST.COM (short username oryx)
I20260927 10:55:14.688886 10080 webserver.cc:460] Webserver started at http://127.1.46.194:46577/ using document root <none> and password file <none>
I20260927 10:55:14.689432 10080 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:14.689533 10080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:14.693482 10080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:55:14.695672 10095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.696889 10080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:14.697033 10080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "173d0f6693a94a4a93a2c10723d23d94"
format_stamp: "Formatted at 2026-09-27 10:55:13 on dist-test-slave-cffh"
server_key: "2e1a4eb5740ff52231592c24252774b4"
server_key_iv: "42af68bac61d78bcdad03107f216a229"
server_key_version: "encryptionkey@0"
I20260927 10:55:14.697572 10080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:14.718677 10080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:14.721854 10080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:14.722066 10080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:14.722728 10080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:14.723595 10080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:14.723640 10080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.723680 10080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:14.723740 10080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.732395 10080 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:44445
I20260927 10:55:14.732486 10208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:44445 every 8 connection(s)
I20260927 10:55:14.733496 10080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.743240 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10080
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.743716 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:34493
--local_ip_for_outbound_sockets=127.1.46.195
--tserver_master_addrs=127.1.46.254:41257
--webserver_port=45307
--webserver_interface=127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:14.744544 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.735435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:39679 (local address 127.1.46.254:41257)
0927 10:55:14.735598 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.735601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.736455 (+ 854us) server_negotiation.cc:407] Connection header received
0927 10:55:14.737715 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.737718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.737772 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.737849 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.739066 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.739592 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.740718 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.740944 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.741026 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.744387 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:39679: BlockingRecv error: recv got EOF from 127.1.46.194:39679 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:55:14.745148 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.735816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:39679)
0927 10:55:14.736283 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.736324 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.737528 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.738011 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.738018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.738385 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.738929 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.738943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.739708 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.739712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.740601 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.740609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.741066 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.741069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.741213 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.741808 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.741823 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.744284 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":129,"threads_started":1}
W20260927 10:55:14.745671 10209 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:41257 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.46.254:41257: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.751978 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.746104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:52633)
0927 10:55:14.746248 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.746264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.746347 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.746640 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.746643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.746781 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.746864 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.746868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.747917 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.747920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.748538 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.748546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.749240 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.749243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.749321 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.749902 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.749913 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.751887 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 10:55:14.752251 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.746224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:52633 (local address 127.1.46.254:41257)
0927 10:55:14.746365 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.746368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.746397 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:55:14.746444 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.746450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.746503 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.746602 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.746973 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.747804 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.748660 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.749101 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.749231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.752083 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:52633: BlockingRecv error: recv got EOF from 127.1.46.194:52633 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.757447 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.752683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:35721)
0927 10:55:14.752802 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.752818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.752893 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.753112 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.753115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.753268 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.753328 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.753333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.754032 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.754038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.754568 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.754615 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.755082 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.755085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.755158 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.755625 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.755635 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.757316 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 10:55:14.757548 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.752783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:35721 (local address 127.1.46.254:41257)
0927 10:55:14.752905 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.752910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.752935 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:55:14.752972 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.752975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.753012 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.753101 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.753437 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.753934 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.754742 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.754977 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.755090 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.757427 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:35721: BlockingRecv error: recv got EOF from 127.1.46.194:35721 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:55:14.757867 10209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:55:14.858430 10212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:14.858788 10212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:14.858836 10212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:14.862886 10212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:14.862949 10212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:14.863008 10212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:14.868350 10212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/krb5kdc/oryx_127.1.46.195.keytab
--principal=oryx/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:34493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=45307
--tserver_master_addrs=127.1.46.254:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:14.869336 10212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:14.870317 10212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:14.879557 10220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:14.879683 10212 server_base.cc:1034] running on GCE node
W20260927 10:55:14.879557 10217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:14.879889 10218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:14.880294 10212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:14.880892 10212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:14.882068 10212 hybrid_clock.cc:648] HybridClock initialized: now 1790506514882040 us; error 42 us; skew 500 ppm
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506514, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:14.885159 10212 init.cc:377] Logged in from keytab as oryx/127.1.46.195@KRBTEST.COM (short username oryx)
I20260927 10:55:14.885958 10212 webserver.cc:460] Webserver started at http://127.1.46.195:45307/ using document root <none> and password file <none>
I20260927 10:55:14.886494 10212 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:14.886615 10212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:14.890259 10212 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:55:14.892259 10227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.893464 10212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:55:14.893586 10212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "c1914f9868b74c1997e4f4540fc7415f"
format_stamp: "Formatted at 2026-09-27 10:55:14 on dist-test-slave-cffh"
server_key: "94a077dbf0ed5f005d1420b06756602e"
server_key_iv: "6ca73c89caa128c68851703ee6d6a5b1"
server_key_version: "encryptionkey@0"
I20260927 10:55:14.894106 10212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:14.922549 10212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:14.925755 10212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:14.925961 10212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:14.926576 10212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:14.927487 10212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:14.927556 10212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.927639 10212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:14.927692 10212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:14.936691 10212 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:34493
I20260927 10:55:14.936758 10340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:34493 every 8 connection(s)
I20260927 10:55:14.937702 10212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:14.941110 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10212
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:55:14.949514 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.939574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:53561 (local address 127.1.46.254:41257)
0927 10:55:14.939724 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.939729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.940598 (+ 869us) server_negotiation.cc:407] Connection header received
0927 10:55:14.941894 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.941897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.941951 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.942034 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.943400 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.944008 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.945356 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.945686 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.945763 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.949393 (+ 3630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:53561: BlockingRecv error: recv got EOF from 127.1.46.195:53561 (error 108)
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":56}
I20260927 10:55:14.950148 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.940015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:53561)
0927 10:55:14.940450 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.940490 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.941677 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.942247 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.942257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.942755 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.943255 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.943275 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.944130 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.944133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.945248 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.945257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.945840 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.945846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.946052 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.946664 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.946680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.949306 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
W20260927 10:55:14.950762 10341 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:41257 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.46.254:41257: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
I20260927 10:55:14.950915 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.942054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.1.46.254:41257)
0927 10:55:14.942299 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.942303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.942341 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:55:14.942378 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.942383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.942431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.942523 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.943286 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.943916 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.944829 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.945173 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.945280 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.946843 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:14.946851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:14.946869 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:14.946900 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:14.948651 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:14.949150 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:14.949154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:14.949159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:14.949272 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:14.949559 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:14.949565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:14.949570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:14.950001 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:14.950202 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:14.950500 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 10:55:14.950758 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.956784 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.951892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:51005)
0927 10:55:14.952005 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.952021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.952091 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.952292 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.952294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.952409 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.952523 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.952530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.953512 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.953516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.954147 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.954155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.954564 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.954611 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.954692 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.955103 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.955114 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.956671 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:55:14.956950 10345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.951955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:51005 (local address 127.1.46.254:41257)
0927 10:55:14.952061 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.952065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.952078 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:14.952151 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.952154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.952190 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.952270 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.952682 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.953387 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.954257 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.954473 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.954551 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.956804 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:51005: BlockingRecv error: recv got EOF from 127.1.46.195:51005 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:14 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:14.961578 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:14.957304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:58161)
0927 10:55:14.957430 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:14.957446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:14.957513 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:14.957684 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:14.957686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:14.957824 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:14.957887 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.957891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.958505 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.958507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.959071 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:14.959078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.959441 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.959443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:14.959495 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.959940 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:14.959951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:14.961497 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:55:14.961704 10345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:14.957363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:58161 (local address 127.1.46.254:41257)
0927 10:55:14.957498 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:14.957501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:14.957512 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:55:14.957569 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:14.957571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:14.957600 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:14.957659 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:14.957974 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.958424 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.959162 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:14.959357 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:14.959428 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:14.961604 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:58161: BlockingRecv error: recv got EOF from 127.1.46.195:58161 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 10:55:14.962002 10341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:55:15 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:15 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:15.573494 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:15.567685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:37981)
0927 10:55:15.567977 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:15.567992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:15.568119 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:15.568395 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:15.568397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:15.568566 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:15.568680 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:15.568685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.569403 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.569405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:15.569996 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:15.570004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.570453 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.570456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:15.570538 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:15.571212 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:15.571235 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:15.573378 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":116,"threads_started":1}
W20260927 10:55:15.573668 10347 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:15.567829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:37981 (local address 127.1.46.254:41257)
0927 10:55:15.568086 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:15.568089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:15.568102 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:15.568212 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:15.568214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:15.568255 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:15.568349 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:15.568778 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.569310 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.570094 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.570324 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.570400 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:15.573501 (+ 3101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:37981: BlockingRecv error: recv got EOF from 127.1.46.193:37981 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":107,"threads_started":1}
Sep 27 10:55:15 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:15 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:15.764035 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:15.758532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:47739)
0927 10:55:15.758885 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:15.758900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:15.759000 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:15.759215 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:15.759217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:15.759392 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:15.759491 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:15.759497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.760202 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.760204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:15.760774 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:15.760781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.761216 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.761219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:15.761285 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:15.761835 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:15.761846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:15.763915 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
W20260927 10:55:15.764140 10347 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:15.758612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:47739 (local address 127.1.46.254:41257)
0927 10:55:15.758751 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:15.758756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:15.758974 (+ 218us) server_negotiation.cc:407] Connection header received
0927 10:55:15.759066 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:15.759068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:15.759111 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:15.759175 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:15.759589 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.760110 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.760891 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.761108 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.761186 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:15.764030 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:47739: BlockingRecv error: recv got EOF from 127.1.46.194:47739 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:55:15 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:15 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:15.968480 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:15.962810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:41013)
0927 10:55:15.963131 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:15.963145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:15.963251 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:15.963496 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:15.963499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:15.963673 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:15.963788 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:15.963795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.964520 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.964523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:15.965175 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:15.965185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.965789 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.965793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:15.965889 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:15.966494 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:15.966505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:15.968366 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":140,"threads_started":1}
W20260927 10:55:15.968639 10347 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:15.962900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:41013 (local address 127.1.46.254:41257)
0927 10:55:15.963048 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:15.963053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:15.963203 (+ 150us) server_negotiation.cc:407] Connection header received
0927 10:55:15.963334 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:15.963337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:15.963381 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:15.963463 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:15.963896 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.964411 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.965315 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:15.965652 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:15.965780 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:15.968503 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:41013: BlockingRecv error: recv got EOF from 127.1.46.195:41013 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 10:55:16 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:16 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:16.579922 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:16.574298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:51605)
0927 10:55:16.574707 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:16.574720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:16.574825 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:16.575087 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:16.575089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:16.575263 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:16.575358 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:16.575363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.576110 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.576113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:16.576656 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:16.576664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.577095 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.577099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:16.577167 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:16.577650 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:16.577662 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:16.579789 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":70,"threads_started":1}
W20260927 10:55:16.580060 10351 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:16.574367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:51605 (local address 127.1.46.254:41257)
0927 10:55:16.574740 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:16.574743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:16.574791 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:55:16.574902 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:16.574905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:16.574946 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:16.575038 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:16.575452 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.576007 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.576750 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.576976 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.577050 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:16.579923 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:51605: BlockingRecv error: recv got EOF from 127.1.46.193:51605 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":216,"threads_started":1}
Sep 27 10:55:16 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:16 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:16.770736 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:16.764976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:49021)
0927 10:55:16.765313 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:16.765330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:16.765469 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:16.765777 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:16.765780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:16.765986 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:16.766078 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:16.766084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.766932 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.766938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:16.767510 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:16.767517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.768028 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.768032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:16.768123 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:16.768585 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:16.768596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:16.770537 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":86,"threads_started":1}
W20260927 10:55:16.770809 10351 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:16.765112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:49021 (local address 127.1.46.254:41257)
0927 10:55:16.765294 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:16.765299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:16.765415 (+ 116us) server_negotiation.cc:407] Connection header received
0927 10:55:16.765576 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:16.765581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:16.765635 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:16.765728 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:16.766208 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.766826 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.767631 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.767906 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.768012 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:16.770719 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:49021: BlockingRecv error: recv got EOF from 127.1.46.194:49021 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 10:55:16 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:16 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:16.974560 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:16.969357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:34647)
0927 10:55:16.969693 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:16.969726 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:16.969845 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:16.970135 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:16.970138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:16.970323 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:16.970435 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:16.970441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.971157 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.971159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:16.971701 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:16.971708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.972151 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.972153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:16.972209 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:16.972770 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:16.972782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:16.974453 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":118,"threads_started":1}
W20260927 10:55:16.974714 10351 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:16.969560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:34647 (local address 127.1.46.254:41257)
0927 10:55:16.969711 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:16.969717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:16.969809 (+ 92us) server_negotiation.cc:407] Connection header received
0927 10:55:16.969940 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:16.969944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:16.969996 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:16.970086 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:16.970534 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.971070 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.971847 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:16.972057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:16.972135 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:16.974603 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:34647: BlockingRecv error: recv got EOF from 127.1.46.195:34647 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 10:55:17 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:17 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:17.586009 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:17.580746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:39253)
0927 10:55:17.581012 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:17.581025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:17.581107 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:17.581312 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:17.581314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:17.581484 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:17.581575 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:17.581580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.582289 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.582291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:17.583015 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:17.583024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.583426 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.583428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:17.583529 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:17.584092 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:17.584104 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:17.585880 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
W20260927 10:55:17.586177 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:17.580801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:39253 (local address 127.1.46.254:41257)
0927 10:55:17.581055 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:17.581059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:17.581071 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:17.581170 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:17.581172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:17.581210 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:17.581282 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:17.581667 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.582200 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.583112 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.583336 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.583415 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:17.586010 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:39253: BlockingRecv error: recv got EOF from 127.1.46.193:39253 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
Sep 27 10:55:17 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:17 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:17.777527 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:17.771563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:51301)
0927 10:55:17.771865 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:17.771880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:17.772022 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:17.772305 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:17.772308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:17.772558 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:17.772698 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:17.772705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.773789 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.773792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:17.774349 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:17.774357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.774934 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.774938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:17.775003 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:17.775557 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:17.775572 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:17.777420 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
W20260927 10:55:17.777725 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:17.771675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:51301 (local address 127.1.46.254:41257)
0927 10:55:17.771812 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:17.771817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:17.772004 (+ 187us) server_negotiation.cc:407] Connection header received
0927 10:55:17.772104 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:17.772106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:17.772169 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:17.772238 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:17.772888 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.773662 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.774500 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.774828 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.774911 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:17.777559 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:51301: BlockingRecv error: recv got EOF from 127.1.46.194:51301 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:55:17 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:17 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:17.980868 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:17.975411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:54855)
0927 10:55:17.975695 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:17.975719 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:17.975830 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:17.976091 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:17.976093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:17.976263 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:17.976351 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:17.976355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.977063 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.977065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:17.977624 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:17.977630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.978055 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.978058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:17.978127 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:17.978651 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:17.978662 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:17.980693 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":121,"threads_started":1}
W20260927 10:55:17.980926 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:17.975489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:54855 (local address 127.1.46.254:41257)
0927 10:55:17.975616 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:17.975620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:17.975800 (+ 180us) server_negotiation.cc:407] Connection header received
0927 10:55:17.975918 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:17.975920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:17.975964 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:17.976051 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:17.976445 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.976968 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.977731 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:17.977959 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:17.978052 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:17.980800 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:54855: BlockingRecv error: recv got EOF from 127.1.46.195:54855 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:55:18 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:18 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:18.598901 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:18.586764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:35333)
0927 10:55:18.587027 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:18.587040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:18.587127 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:18.587385 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:18.587388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:18.587617 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:18.587751 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:18.587777 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.588797 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.588802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:18.589368 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:18.589375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.589845 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.589848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:18.589935 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:18.596731 (+ 6796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:18.596744 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:18.598773 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
W20260927 10:55:18.599148 10359 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:18.586849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:35333 (local address 127.1.46.254:41257)
0927 10:55:18.587088 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:18.587091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:18.587105 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:55:18.587213 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:18.587216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:18.587257 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:18.587344 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:18.587913 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.588704 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.589489 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.589735 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.589843 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:18.599007 (+ 9164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:35333: BlockingRecv error: recv got EOF from 127.1.46.193:35333 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
Sep 27 10:55:18 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:18 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:18.783957 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:18.778284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:35303)
0927 10:55:18.778686 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:18.778705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:18.778806 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:18.779136 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:18.779139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:18.779360 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:18.779503 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:18.779511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.780306 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.780309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:18.780842 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:18.780849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.781342 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.781345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:18.781405 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:18.781844 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:18.781854 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:18.783844 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":115,"threads_started":1}
W20260927 10:55:18.784210 10359 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:18.778418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:35303 (local address 127.1.46.254:41257)
0927 10:55:18.778648 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:18.778652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:18.778790 (+ 138us) server_negotiation.cc:407] Connection header received
0927 10:55:18.778940 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:18.778943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:18.779015 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:18.779099 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:18.779623 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.780213 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.780946 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.781252 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.781325 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:18.783991 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:35303: BlockingRecv error: recv got EOF from 127.1.46.194:35303 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 10:55:18 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:18 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:18.987530 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:18.981621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:42939)
0927 10:55:18.981873 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:18.981886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:18.981982 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:18.982260 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:18.982262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:18.982430 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:18.982524 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:18.982529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.983587 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.983593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:18.984227 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:18.984235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.984696 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.984700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:18.984783 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:18.985235 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:18.985246 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:18.987357 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
W20260927 10:55:18.987569 10359 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:18.981746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:42939 (local address 127.1.46.254:41257)
0927 10:55:18.981913 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:18.981917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:18.981930 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:18.982077 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:18.982081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:18.982138 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:18.982203 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:18.982738 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.983469 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.984344 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:18.984587 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:18.984693 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:18.987473 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:42939: BlockingRecv error: recv got EOF from 127.1.46.195:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 10:55:19 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:19 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:19.604936 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:19.599755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:47209)
0927 10:55:19.600042 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:19.600056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:19.600152 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:19.600355 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:19.600357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:19.600529 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:19.600617 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:19.600622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.601354 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.601356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:19.601948 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:19.601956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.602394 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.602397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:19.602453 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:19.603002 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:19.603014 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:19.604817 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":122,"threads_started":1}
W20260927 10:55:19.605091 10363 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:19.599804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:47209 (local address 127.1.46.254:41257)
0927 10:55:19.600032 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:19.600036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:19.600126 (+ 90us) server_negotiation.cc:407] Connection header received
0927 10:55:19.600226 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:19.600229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:19.600268 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:19.600339 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:19.600700 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.601251 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.602075 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.602298 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.602377 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:19.604956 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:47209: BlockingRecv error: recv got EOF from 127.1.46.193:47209 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
Sep 27 10:55:19 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:19 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:19.790179 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:19.784665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:48723)
0927 10:55:19.784986 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:19.785003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:19.785133 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:19.785413 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:19.785416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:19.785631 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:19.785719 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:19.785724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.786494 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.786499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:19.787101 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:19.787108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.787582 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.787586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:19.787676 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:19.788238 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:19.788250 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:19.790012 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
W20260927 10:55:19.790230 10363 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:19.784801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:48723 (local address 127.1.46.254:41257)
0927 10:55:19.784955 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:19.784960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:19.785080 (+ 120us) server_negotiation.cc:407] Connection header received
0927 10:55:19.785235 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:19.785240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:19.785290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:19.785395 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:19.785839 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.786385 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.787222 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.787472 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.787582 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:19.790126 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:48723: BlockingRecv error: recv got EOF from 127.1.46.194:48723 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 10:55:19 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:19 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:19.994397 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:19.988334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:58311)
0927 10:55:19.988597 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:19.988610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:19.988713 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:19.988965 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:19.988986 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:19.989159 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:19.989245 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:19.989262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.990281 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.990287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:19.991033 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:19.991045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.991588 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.991592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:19.991682 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:19.992221 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:19.992233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:19.994210 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
W20260927 10:55:19.994431 10363 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:19.988385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:58311 (local address 127.1.46.254:41257)
0927 10:55:19.988526 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:19.988530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:19.988694 (+ 164us) server_negotiation.cc:407] Connection header received
0927 10:55:19.988790 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:19.988792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:19.988835 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:19.988925 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:19.989423 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.990147 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.991178 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:19.991461 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:19.991587 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:19.994329 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:58311: BlockingRecv error: recv got EOF from 127.1.46.195:58311 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:55:20 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:20 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:20.611552 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:20.605793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:50453)
0927 10:55:20.606048 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:20.606071 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:20.606158 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:20.606458 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:20.606461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:20.606720 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:20.606808 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:20.606814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.607685 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.607692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:20.608292 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:20.608300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.608787 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.608790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:20.608877 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:20.609359 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:20.609370 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:20.611354 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":103,"threads_started":1}
W20260927 10:55:20.611620 10367 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:20.605856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:50453 (local address 127.1.46.254:41257)
0927 10:55:20.606102 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:20.606107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:20.606120 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:20.606266 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:20.606270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:20.606324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:20.606437 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:20.606934 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.607581 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.608415 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.608678 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.608785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:20.611492 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:50453: BlockingRecv error: recv got EOF from 127.1.46.193:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
Sep 27 10:55:20 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:20 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:20.796046 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:20.790978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:58551)
0927 10:55:20.791248 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:20.791271 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:20.791357 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:20.791593 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:20.791595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:20.791773 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:20.791863 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:20.791868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.792593 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.792595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:20.793148 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:20.793155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.793562 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.793565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:20.793620 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:20.794154 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:20.794165 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:20.795926 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
W20260927 10:55:20.796204 10367 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:20.791165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:58551 (local address 127.1.46.254:41257)
0927 10:55:20.791295 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:20.791298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:20.791313 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:55:20.791447 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:20.791449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:20.791491 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:20.791560 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:20.791963 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.792494 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.793249 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.793474 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.793551 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:20.796053 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:58551: BlockingRecv error: recv got EOF from 127.1.46.194:58551 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:55:20 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:20 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:21.000793 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:20.995144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:47369)
0927 10:55:20.995523 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:20.995535 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:20.995661 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:20.996014 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:20.996018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:20.996229 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:20.996313 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:20.996318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.997081 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.997087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:20.997736 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:20.997744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.998204 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.998207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:20.998295 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:20.998874 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:20.998886 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:21.000666 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":98,"threads_started":1}
W20260927 10:55:21.000962 10367 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:20.995266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:47369 (local address 127.1.46.254:41257)
0927 10:55:20.995443 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:20.995448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:20.995606 (+ 158us) server_negotiation.cc:407] Connection header received
0927 10:55:20.995770 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:20.995775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:20.995829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:20.995985 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:20.996431 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.996972 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.997859 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:20.998088 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:20.998192 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:21.000826 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:47369: BlockingRecv error: recv got EOF from 127.1.46.195:47369 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 10:55:21 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:21 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:21.618314 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:21.612343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:47677)
0927 10:55:21.612734 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:21.612748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:21.612849 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:21.613174 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:21.613177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:21.613390 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:21.613500 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:21.613506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.614343 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.614346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:21.615002 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:21.615016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.615630 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.615632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:21.615733 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:21.616377 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:21.616391 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:21.618202 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":103,"threads_started":1}
W20260927 10:55:21.618501 10371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:21.612470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:47677 (local address 127.1.46.254:41257)
0927 10:55:21.612731 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:21.612735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:21.612855 (+ 120us) server_negotiation.cc:407] Connection header received
0927 10:55:21.612964 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:21.612967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:21.613030 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:21.613106 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:21.613609 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.614241 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.615123 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.615503 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.615630 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:21.618322 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:47677: BlockingRecv error: recv got EOF from 127.1.46.193:47677 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":64,"threads_started":1}
Sep 27 10:55:21 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:21 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:21.803057 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:21.796815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:58089)
0927 10:55:21.797179 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:21.797191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:21.797329 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:21.797658 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:21.797660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:21.797834 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:21.797922 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:21.797927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.798885 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.798888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:21.799713 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:21.799722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.800278 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.800282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:21.800369 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:21.800861 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:21.800875 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:21.802851 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":79,"threads_started":1}
W20260927 10:55:21.803098 10371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:21.796939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:58089 (local address 127.1.46.254:41257)
0927 10:55:21.797100 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:21.797105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:21.797267 (+ 162us) server_negotiation.cc:407] Connection header received
0927 10:55:21.797493 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:21.797495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:21.797542 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:21.797611 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:21.798056 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.798755 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.799827 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:21.800132 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:21.800277 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:21.802989 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:58089: BlockingRecv error: recv got EOF from 127.1.46.194:58089 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 10:55:21 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:22 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:22.007503 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:22.001730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:51717)
0927 10:55:22.002134 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:22.002147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:22.002245 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:22.002482 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:22.002484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:22.002682 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:22.002774 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:22.002778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.003508 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.003511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:22.004029 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:22.004036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.004502 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.004506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:22.004603 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:22.005193 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:22.005206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:22.007318 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
W20260927 10:55:22.007540 10371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:22.001805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:51717 (local address 127.1.46.254:41257)
0927 10:55:22.001970 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:22.001977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:22.002224 (+ 247us) server_negotiation.cc:407] Connection header received
0927 10:55:22.002320 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:22.002322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:22.002370 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:22.002455 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:22.002871 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.003387 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.004133 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.004353 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.004501 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:22.007445 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:51717: BlockingRecv error: recv got EOF from 127.1.46.195:51717 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 10:55:22 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:22 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:22.624485 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:22.619035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:45307)
0927 10:55:22.619358 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:22.619377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:22.619490 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:22.619734 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:22.619736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:22.619908 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:22.619996 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:22.620001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.620757 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.620759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:22.621320 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:22.621328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.621733 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.621735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:22.621793 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:22.622283 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:22.622294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:22.624313 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
W20260927 10:55:22.624630 10375 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:22.619242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:45307 (local address 127.1.46.254:41257)
0927 10:55:22.619469 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:22.619473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:22.619487 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:55:22.619579 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:22.619582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:22.619622 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:22.619697 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:22.620121 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.620658 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.621415 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.621639 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.621710 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:22.624476 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:45307: BlockingRecv error: recv got EOF from 127.1.46.193:45307 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
Sep 27 10:55:22 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:22 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:22.810055 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:22.803821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:36279)
0927 10:55:22.804098 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:22.804112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:22.804217 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:22.804551 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:22.804553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:22.804743 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:22.804837 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:22.804842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.806025 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.806030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:22.806751 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:22.806762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.807276 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.807280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:22.807360 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:22.807972 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:22.807987 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:22.809892 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
W20260927 10:55:22.810153 10375 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:22.803909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:36279 (local address 127.1.46.254:41257)
0927 10:55:22.804089 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:22.804093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:22.804188 (+ 95us) server_negotiation.cc:407] Connection header received
0927 10:55:22.804340 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:22.804343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:22.804396 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:22.804489 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:22.804977 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.805888 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.806878 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:22.807122 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:22.807270 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:22.810040 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:36279: BlockingRecv error: recv got EOF from 127.1.46.194:36279 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 10:55:23 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:23 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:23.013921 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:23.008203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:47471)
0927 10:55:23.008500 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:23.008514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:23.008615 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:23.008839 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:23.008842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:23.009029 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:23.009121 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:23.009126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.009873 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.009875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:23.010438 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:23.010445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.010910 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.010913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:23.010970 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:23.011553 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:23.011572 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:23.013810 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
W20260927 10:55:23.014111 10375 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:23.008240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:47471 (local address 127.1.46.254:41257)
0927 10:55:23.008405 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:23.008410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:23.008591 (+ 181us) server_negotiation.cc:407] Connection header received
0927 10:55:23.008683 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:23.008686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:23.008733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:23.008807 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:23.009251 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.009774 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.010544 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.010807 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.010885 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:23.013960 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:47471: BlockingRecv error: recv got EOF from 127.1.46.195:47471 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 10:55:23 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:23 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:23.634264 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:23.625257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:45425)
0927 10:55:23.625670 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:23.625689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:23.625800 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:23.626080 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:23.626083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:23.626279 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:23.626386 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:23.626392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.627171 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.627173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:23.627812 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:23.627820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.628266 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.628269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:23.628333 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:23.631864 (+ 3531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:23.631877 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:23.634150 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
W20260927 10:55:23.634430 10379 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:23.625409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:45425 (local address 127.1.46.254:41257)
0927 10:55:23.625753 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:23.625757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:23.625770 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:23.625888 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:23.625890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:23.625931 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:23.626006 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:23.626504 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.627078 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.627928 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.628168 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.628248 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:23.634268 (+ 6020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:45425: BlockingRecv error: recv got EOF from 127.1.46.193:45425 (error 108)
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
Sep 27 10:55:23 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:23 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:23.816929 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:23.810867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:54779)
0927 10:55:23.811205 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:23.811223 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:23.811353 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:23.811620 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:23.811624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:23.811845 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:23.811945 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:23.811951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.812707 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.812710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:23.813460 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:23.813469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.813918 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.813921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:23.813987 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:23.814510 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:23.814524 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:23.816791 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
W20260927 10:55:23.817035 10379 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:23.810929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:54779 (local address 127.1.46.254:41257)
0927 10:55:23.811076 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:23.811080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:23.811322 (+ 242us) server_negotiation.cc:407] Connection header received
0927 10:55:23.811431 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:23.811433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:23.811475 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:23.811543 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:23.812082 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.812601 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.813579 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:23.813792 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:23.813868 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:23.816910 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:54779: BlockingRecv error: recv got EOF from 127.1.46.194:54779 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 10:55:24 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:24 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:24.020282 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:24.014716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:35323)
0927 10:55:24.014988 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:24.015025 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:24.015121 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:24.015366 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:24.015368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:24.015547 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:24.015642 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:24.015647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.016362 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.016365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:24.016915 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:24.016922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.017367 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.017370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:24.017429 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:24.017978 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:24.018057 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:24.020122 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
W20260927 10:55:24.020315 10379 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:24.014826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:35323 (local address 127.1.46.254:41257)
0927 10:55:24.014961 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:24.014965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:24.015093 (+ 128us) server_negotiation.cc:407] Connection header received
0927 10:55:24.015199 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:24.015201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:24.015244 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:24.015316 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:24.015740 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.016249 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.017024 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.017258 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.017337 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:24.020216 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:35323: BlockingRecv error: recv got EOF from 127.1.46.195:35323 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:55:24 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:24 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:24.640954 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:24.635107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:34719)
0927 10:55:24.635387 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:24.635403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:24.635497 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:24.635731 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:24.635733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:24.635907 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:24.636011 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:24.636016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.636867 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.636870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:24.637452 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:24.637460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.637902 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.637905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:24.637967 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:24.638482 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:24.638494 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:24.640786 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
W20260927 10:55:24.641121 10383 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:24.635243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:34719 (local address 127.1.46.254:41257)
0927 10:55:24.635509 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:24.635512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:24.635528 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:55:24.635569 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:24.635571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:24.635612 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:24.635695 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:24.636184 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.636749 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.637569 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.637792 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.637872 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:24.640967 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:34719: BlockingRecv error: recv got EOF from 127.1.46.193:34719 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":125,"threads_started":1}
Sep 27 10:55:24 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:24 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:24.823523 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:24.817758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:38179)
0927 10:55:24.818045 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:24.818059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:24.818153 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:24.818416 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:24.818419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:24.818694 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:24.818791 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:24.818797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.819567 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.819569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:24.820144 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:24.820152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.820569 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.820572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:24.820629 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:24.821169 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:24.821181 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:24.823353 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260927 10:55:24.823596 10383 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:24.817885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:38179 (local address 127.1.46.254:41257)
0927 10:55:24.818039 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:24.818048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:24.818133 (+ 85us) server_negotiation.cc:407] Connection header received
0927 10:55:24.818220 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:24.818222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:24.818263 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:24.818337 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:24.818920 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.819466 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.820257 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:24.820476 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:24.820561 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:24.823475 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:38179: BlockingRecv error: recv got EOF from 127.1.46.194:38179 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:55:25 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:25 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:25.026572 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:25.021047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:44755)
0927 10:55:25.021349 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:25.021365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:25.021505 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:25.021771 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:25.021774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:25.021989 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:25.022076 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:25.022082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.022906 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.022913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:25.023470 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:25.023478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.023936 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.023939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:25.024023 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:25.024599 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:25.024611 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:25.026407 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
W20260927 10:55:25.026662 10383 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:25.021209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:44755 (local address 127.1.46.254:41257)
0927 10:55:25.021349 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:25.021355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:25.021449 (+ 94us) server_negotiation.cc:407] Connection header received
0927 10:55:25.021602 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:25.021608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:25.021664 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:25.021750 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:25.022199 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.022801 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.023586 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.023810 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.023933 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:25.026520 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:44755: BlockingRecv error: recv got EOF from 127.1.46.195:44755 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 10:55:25 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:25 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:25.647094 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:25.641820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:48113)
0927 10:55:25.642099 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:25.642117 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:25.642205 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:25.642444 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:25.642446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:25.642651 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:25.642741 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:25.642745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.643505 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.643508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:25.644079 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:25.644087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.644491 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.644493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:25.644552 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:25.645130 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:25.645142 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:25.646946 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
W20260927 10:55:25.647258 10387 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:25.641877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:48113 (local address 127.1.46.254:41257)
0927 10:55:25.642180 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:25.642184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:25.642197 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:25.642291 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:25.642293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:25.642334 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:25.642410 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:25.642871 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.643401 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.644187 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.644401 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.644476 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:25.647115 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:48113: BlockingRecv error: recv got EOF from 127.1.46.193:48113 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":62,"threads_started":1}
Sep 27 10:55:25 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:25 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:25.829680 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:25.824241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:53845)
0927 10:55:25.824508 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:25.824529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:25.824627 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:25.824931 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:25.824933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:25.825130 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:25.825225 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:25.825230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.825969 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.825972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:25.826543 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:25.826551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.827021 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.827024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:25.827122 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:25.827616 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:25.827628 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:25.829490 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
W20260927 10:55:25.829807 10387 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:25.824369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:53845 (local address 127.1.46.254:41257)
0927 10:55:25.824506 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:25.824510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:25.824602 (+ 92us) server_negotiation.cc:407] Connection header received
0927 10:55:25.824720 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:25.824735 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:25.824793 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:25.824889 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:25.825328 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.825866 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.826689 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:25.826916 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:25.827007 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:25.829685 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:53845: BlockingRecv error: recv got EOF from 127.1.46.194:53845 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 10:55:26 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:26 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:26.032855 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:26.027347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:38707)
0927 10:55:26.027665 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:26.027678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:26.027814 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:26.028084 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:26.028087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:26.028289 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:26.028378 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:26.028384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.029169 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.029175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:26.029773 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:26.029781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.030248 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.030251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:26.030335 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:26.030834 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:26.030845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:26.032672 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":80,"threads_started":1}
W20260927 10:55:26.032898 10387 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:26.027453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:38707 (local address 127.1.46.254:41257)
0927 10:55:26.027603 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:26.027609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:26.027749 (+ 140us) server_negotiation.cc:407] Connection header received
0927 10:55:26.027911 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:26.027916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:26.027966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:26.028069 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:26.028493 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.029029 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.029890 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.030139 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.030247 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:26.032795 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:38707: BlockingRecv error: recv got EOF from 127.1.46.195:38707 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 10:55:26 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:26 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:26.653175 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:26.647898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:32877)
0927 10:55:26.648223 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:26.648236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:26.648322 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:26.648555 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:26.648557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:26.648731 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:26.648827 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:26.648832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.649544 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.649546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:26.650171 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:26.650179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.650668 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.650674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:26.650766 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:26.651341 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:26.651352 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:26.653064 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
W20260927 10:55:26.653322 10391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:26.648021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:32877 (local address 127.1.46.254:41257)
0927 10:55:26.648272 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:26.648276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:26.648289 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:55:26.648407 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:26.648409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:26.648449 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:26.648522 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:26.648921 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.649438 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.650291 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.650536 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.650701 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:26.653189 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:32877: BlockingRecv error: recv got EOF from 127.1.46.193:32877 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":68,"threads_started":1}
Sep 27 10:55:26 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:26 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:26.836794 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:26.830509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:38877)
0927 10:55:26.830952 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:26.830965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:26.831108 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:26.831412 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:26.831415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:26.831645 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:26.831738 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:26.831744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.832620 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.832623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:26.833366 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:26.833379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.834010 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.834017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:26.834127 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:26.834683 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:26.834694 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:26.836686 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
W20260927 10:55:26.836937 10391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:26.830682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:38877 (local address 127.1.46.254:41257)
0927 10:55:26.830881 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:26.830886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:26.831047 (+ 161us) server_negotiation.cc:407] Connection header received
0927 10:55:26.831229 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:26.831234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:26.831292 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:26.831389 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:26.831864 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.832409 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.833512 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:26.833874 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:26.833996 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:26.836817 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:38877: BlockingRecv error: recv got EOF from 127.1.46.194:38877 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 10:55:27 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:27 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:27.040091 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:27.033751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:54397)
0927 10:55:27.034128 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:27.034143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:27.034305 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:27.034620 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:27.034624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:27.034851 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:27.034944 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:27.034950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.035754 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.035761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:27.036509 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:27.036519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.037111 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.037114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:27.037211 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:27.037819 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:27.037832 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:27.039866 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":93,"threads_started":1}
W20260927 10:55:27.040130 10391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:27.033879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:54397 (local address 127.1.46.254:41257)
0927 10:55:27.034041 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:27.034047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:27.034228 (+ 181us) server_negotiation.cc:407] Connection header received
0927 10:55:27.034413 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:27.034418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:27.034474 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:27.034611 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:27.035069 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.035638 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.036646 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.036977 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.037101 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:27.040016 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:54397: BlockingRecv error: recv got EOF from 127.1.46.195:54397 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 10:55:27 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:27 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:27.659080 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:27.653959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:50459)
0927 10:55:27.654226 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:27.654239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:27.654326 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:27.654548 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:27.654550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:27.654753 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:27.654846 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:27.654852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.655594 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.655596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:27.656138 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:27.656144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.656556 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.656559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:27.656616 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:27.657159 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:27.657172 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:27.658969 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
W20260927 10:55:27.659272 10395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:27.654052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:50459 (local address 127.1.46.254:41257)
0927 10:55:27.654294 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:27.654297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:27.654309 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:27.654393 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:27.654395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:27.654435 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:27.654512 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:27.654944 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.655504 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.656248 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.656467 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.656539 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:27.659093 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:50459: BlockingRecv error: recv got EOF from 127.1.46.193:50459 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
Sep 27 10:55:27 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:27 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:27.843732 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:27.837616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:35229)
0927 10:55:27.837906 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:27.837922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:27.838033 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:27.838298 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:27.838301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:27.838502 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:27.838705 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:27.838714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.839450 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.839453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:27.840108 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:27.840117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.840526 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.840529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:27.840608 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:27.841252 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:27.841266 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:27.843606 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
W20260927 10:55:27.843840 10395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:27.837746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:35229 (local address 127.1.46.254:41257)
0927 10:55:27.837890 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:27.837894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:27.837997 (+ 103us) server_negotiation.cc:407] Connection header received
0927 10:55:27.838134 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:27.838136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:27.838177 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:27.838241 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:27.838833 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.839348 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.840217 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:27.840426 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:27.840496 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:27.843734 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:35229: BlockingRecv error: recv got EOF from 127.1.46.194:35229 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:55:28 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:28 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:28.046290 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:28.040853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:38177)
0927 10:55:28.041169 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:28.041181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:28.041310 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:28.041578 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:28.041581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:28.041822 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:28.041928 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:28.041933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.042748 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.042754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:28.043327 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:28.043335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.043790 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.043793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:28.043879 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:28.044373 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:28.044384 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:28.046172 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260927 10:55:28.046527 10395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:28.040907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:38177 (local address 127.1.46.254:41257)
0927 10:55:28.041076 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:28.041080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:28.041259 (+ 179us) server_negotiation.cc:407] Connection header received
0927 10:55:28.041412 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:28.041418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:28.041471 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:28.041557 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:28.042041 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.042645 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.043442 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.043680 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.043788 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:28.046356 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:38177: BlockingRecv error: recv got EOF from 127.1.46.195:38177 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 10:55:28 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:28 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:28.671247 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:28.659868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:58475)
0927 10:55:28.660131 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:28.660144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:28.660228 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:28.660433 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:28.660435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:28.660604 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:28.660693 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:28.660698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.661423 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.661425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:28.661971 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:28.661978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.662367 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.662369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:28.662426 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:28.669241 (+ 6815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:28.669271 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:28.671132 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
W20260927 10:55:28.671399 10399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:28.659963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:58475 (local address 127.1.46.254:41257)
0927 10:55:28.660235 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:28.660238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:28.660250 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:55:28.660285 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:28.660288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:28.660327 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:28.660403 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:28.660822 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.661334 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.662065 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.662278 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.662351 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:28.671264 (+ 8913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:58475: BlockingRecv error: recv got EOF from 127.1.46.193:58475 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
Sep 27 10:55:28 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:28 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:28.849638 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:28.844500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:42715)
0927 10:55:28.844804 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:28.844822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:28.844966 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:28.845212 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:28.845214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:28.845401 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:28.845490 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:28.845495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.846175 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.846177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:28.846761 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:28.846769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.847172 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.847175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:28.847231 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:28.847703 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:28.847714 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:28.849515 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
W20260927 10:55:28.849956 10399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:28.844626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:42715 (local address 127.1.46.254:41257)
0927 10:55:28.844782 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:28.844787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:28.844940 (+ 153us) server_negotiation.cc:407] Connection header received
0927 10:55:28.845068 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:28.845070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:28.845116 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:28.845180 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:28.845586 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.846090 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.846860 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:28.847081 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:28.847156 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:28.849688 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:42715: BlockingRecv error: recv got EOF from 127.1.46.194:42715 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:55:29 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:29 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.195@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:29.053381 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:29.047232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.195:37483)
0927 10:55:29.047496 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:29.047509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:29.047601 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:29.047892 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:29.047895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:29.048101 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:29.048194 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:29.048199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.049393 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.049399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:29.050048 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:29.050058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.050653 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.050658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:29.050758 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:29.051279 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:29.051293 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:29.053261 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
W20260927 10:55:29.053514 10399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:29.047288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:37483 (local address 127.1.46.254:41257)
0927 10:55:29.047410 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:29.047414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:29.047574 (+ 160us) server_negotiation.cc:407] Connection header received
0927 10:55:29.047665 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:29.047667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:29.047708 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:29.047788 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:29.048415 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.049249 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.050217 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.050490 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.050656 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:29.053373 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.195:37483: BlockingRecv error: recv got EOF from 127.1.46.195:37483 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:55:29 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:29 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.193@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:29.677983 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:29.672223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.193:57315)
0927 10:55:29.672506 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:29.672519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:29.672613 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:29.672864 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:29.672866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:29.673038 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:29.673128 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:29.673133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.673847 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.673849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:29.674382 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:29.674388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.675065 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.675069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:29.675148 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:29.675858 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:29.675881 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:29.677864 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
W20260927 10:55:29.678193 10403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:29.672274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:57315 (local address 127.1.46.254:41257)
0927 10:55:29.672530 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:29.672533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:29.672594 (+ 61us) server_negotiation.cc:407] Connection header received
0927 10:55:29.672697 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:29.672699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:29.672750 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:29.672828 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:29.673230 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.673759 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.674489 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.674938 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.675045 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:29.678001 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.193:57315: BlockingRecv error: recv got EOF from 127.1.46.193:57315 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
Sep 27 10:55:29 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:55:29 dist-test-slave-cffh krb5kdc[9345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.46.194@KRBTEST.COM for oryx/127.1.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:55:29.856689 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:29.850700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:41257 (local address 127.1.46.194:48385)
0927 10:55:29.851010 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:29.851025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:29.851113 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:29.851465 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:29.851468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:29.851716 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:29.851816 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:29.851822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.852628 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.852631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:29.853273 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:29.853282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.853760 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.853763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:29.853828 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:29.854344 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:29.854359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:29.856497 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.46.254:41257: Server oryx/127.1.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":118,"threads_started":1}
W20260927 10:55:29.856726 10403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:29.850776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:48385 (local address 127.1.46.254:41257)
0927 10:55:29.851043 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:29.851047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:29.851065 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:55:29.851278 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:29.851281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:29.851327 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:29.851407 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:29.851958 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.852513 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.853411 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:29.853633 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:29.853720 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:29.856623 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.46.194:48385: BlockingRecv error: recv got EOF from 127.1.46.194:48385 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
I20260927 10:55:29.951625 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 9928
I20260927 10:55:29.957088 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10080
I20260927 10:55:29.962020 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10212
I20260927 10:55:29.967152 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 9858
2026-09-27T10:55:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17920 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T10:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:29Z Disabled control of system clock
I20260927 10:55:29.995923 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:39787
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:55:30.109833 10408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:30.110136 10408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:30.110185 10408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:30.114069 10408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:30.114118 10408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:30.114136 10408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:30.114161 10408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:30.114179 10408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:30.118669 10408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:39787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:30.119524 10408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:30.120810 10408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:30.129025 10414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:30.129182 10408 server_base.cc:1034] running on GCE node
W20260927 10:55:30.129036 10413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.129026 10416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:30.129900 10408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:30.131031 10408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:30.132232 10408 hybrid_clock.cc:648] HybridClock initialized: now 1790506530132221 us; error 34 us; skew 500 ppm
I20260927 10:55:30.134017 10408 webserver.cc:460] Webserver started at http://127.1.46.254:36633/ using document root <none> and password file <none>
I20260927 10:55:30.134688 10408 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:30.134784 10408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:30.135021 10408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:30.137179 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "7e7ed721222c4599a74e763dff79e083"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "405f1820fbc822685b8abf27c1aefcf8"
server_key_iv: "4cf1021cec55c3cfee3a27c6a520d7bf"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.137763 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "7e7ed721222c4599a74e763dff79e083"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "405f1820fbc822685b8abf27c1aefcf8"
server_key_iv: "4cf1021cec55c3cfee3a27c6a520d7bf"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.141408 10408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:55:30.143790 10422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.145000 10408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:30.145141 10408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "7e7ed721222c4599a74e763dff79e083"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "405f1820fbc822685b8abf27c1aefcf8"
server_key_iv: "4cf1021cec55c3cfee3a27c6a520d7bf"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.145264 10408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:30.167783 10408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:30.168484 10408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:30.168694 10408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:30.176071 10408 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:39787
I20260927 10:55:30.176087 10474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:39787 every 8 connection(s)
I20260927 10:55:30.177124 10408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:30.179968 10475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:30.181254 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10408
I20260927 10:55:30.181344 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:55:30.181658 1211 external_mini_cluster.cc:1428] Setting key 6a75320ad1e2084271a0950deb84d6d2
I20260927 10:55:30.185927 10475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083: Bootstrap starting.
I20260927 10:55:30.188506 10475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:30.189342 10475 log.cc:824] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:30.190814 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:30.183043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.1.46.254:39787)
0927 10:55:30.183666 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:30.183689 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:30.183723 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:55:30.184330 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:30.184346 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:30.184712 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:30.185032 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:30.185347 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.186336 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.187541 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.188001 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.188118 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:30.189026 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:30.189031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:30.189043 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:30.189418 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:30.189505 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:30.189537 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:30.189610 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:30.189797 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:55:30.190055 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":161,"threads_started":1}
I20260927 10:55:30.191517 10475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083: No bootstrap required, opened a new log
I20260927 10:55:30.195497 10475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7ed721222c4599a74e763dff79e083" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39787 } }
I20260927 10:55:30.195696 10475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:30.195755 10475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7ed721222c4599a74e763dff79e083, State: Initialized, Role: FOLLOWER
I20260927 10:55:30.196144 10475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7ed721222c4599a74e763dff79e083" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39787 } }
I20260927 10:55:30.196275 10475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:30.196313 10475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:30.196378 10475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:30.197130 10475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7ed721222c4599a74e763dff79e083" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39787 } }
I20260927 10:55:30.197396 10475 leader_election.cc:302] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e7ed721222c4599a74e763dff79e083; no voters:
I20260927 10:55:30.197623 10475 leader_election.cc:288] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:30.197800 10480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:30.198079 10480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7ed721222c4599a74e763dff79e083, State: Running, Role: LEADER
I20260927 10:55:30.198441 10480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7ed721222c4599a74e763dff79e083" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39787 } }
I20260927 10:55:30.198916 10475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:30.200140 10481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e7ed721222c4599a74e763dff79e083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7ed721222c4599a74e763dff79e083" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39787 } } }
I20260927 10:55:30.200271 10481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:30.200500 10482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e7ed721222c4599a74e763dff79e083. Latest consensus state: current_term: 1 leader_uuid: "7e7ed721222c4599a74e763dff79e083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7ed721222c4599a74e763dff79e083" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39787 } } }
I20260927 10:55:30.200652 10482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:30.200759 10490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:30.204208 10490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:30.209224 10490 catalog_manager.cc:1347] Generated new cluster ID: d257af3b3a3248818d3a779210247c0d
I20260927 10:55:30.209287 10490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:30.222667 10490 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:30.223785 10490 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:30.230994 10490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e7ed721222c4599a74e763dff79e083: Generated new TSK 0
I20260927 10:55:30.231721 10490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:55:30.237005 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39787
--builtin_ntp_servers=127.1.46.212:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 10:55:30.347642 10499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:30.347883 10499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:30.347918 10499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:30.351971 10499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:30.352053 10499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:30.352134 10499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:30.357244 10499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:30.358637 10499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:30.360412 10499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:30.370203 10507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.370220 10505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.370211 10504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:30.370380 10499 server_base.cc:1034] running on GCE node
I20260927 10:55:30.370870 10499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:30.371452 10499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:30.372610 10499 hybrid_clock.cc:648] HybridClock initialized: now 1790506530372595 us; error 27 us; skew 500 ppm
I20260927 10:55:30.374287 10499 webserver.cc:460] Webserver started at http://127.1.46.193:43465/ using document root <none> and password file <none>
I20260927 10:55:30.374873 10499 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:30.374934 10499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:30.375090 10499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:30.377005 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "a7cd23a146784eabbd5e7a95d21e5d75"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "5ffd0350b41d5c56cb55a36637c859d9"
server_key_iv: "713218b921cbaaff42f58e9afeff5853"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.377485 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "a7cd23a146784eabbd5e7a95d21e5d75"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "5ffd0350b41d5c56cb55a36637c859d9"
server_key_iv: "713218b921cbaaff42f58e9afeff5853"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.380939 10499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:55:30.383279 10513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.384409 10499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:30.384497 10499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "a7cd23a146784eabbd5e7a95d21e5d75"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "5ffd0350b41d5c56cb55a36637c859d9"
server_key_iv: "713218b921cbaaff42f58e9afeff5853"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.384577 10499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:30.396718 10499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:30.397300 10499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:30.397450 10499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:30.397953 10499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:30.398959 10499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:30.399011 10499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.399106 10499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:30.399148 10499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.408983 10499 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:33939
I20260927 10:55:30.409065 10626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:33939 every 8 connection(s)
I20260927 10:55:30.409924 10499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:30.413134 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10499
I20260927 10:55:30.413269 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:30.413563 1211 external_mini_cluster.cc:1428] Setting key 75d7297a9e37767ce17f894c1de273f3
I20260927 10:55:30.415591 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39787
--builtin_ntp_servers=127.1.46.212:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:55:30.419343 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:30.411770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:60467 (local address 127.1.46.254:39787)
0927 10:55:30.411920 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:30.411925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:30.413003 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 10:55:30.414616 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:30.414620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:30.414675 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:30.414787 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:30.415700 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.416235 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.417329 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.417563 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.417673 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:30.418847 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:30.418858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:30.418860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:30.418870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:30.418888 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:30.418895 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:30.418976 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:30.419132 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:55:30.419196 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:55:30.420145 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:30.412219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39787 (local address 127.1.46.193:60467)
0927 10:55:30.412851 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:30.412889 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:30.414391 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:30.414990 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:30.414999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:30.415100 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:30.415569 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:30.415581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.416369 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.416380 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:30.417209 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:30.417218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.417718 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.417722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:30.417878 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:30.418536 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:55:30.418538 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:30.418620 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:55:30.418686 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:30.418978 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:30.418989 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:30.418997 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:55:30.419120 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 10:55:30.419368 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260927 10:55:30.421396 10627 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39787
I20260927 10:55:30.421617 10627 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:30.422082 10627 heartbeater.cc:511] Master 127.1.46.254:39787 requested a full tablet report, sending...
I20260927 10:55:30.423506 10439 ts_manager.cc:194] Registered new tserver with Master: a7cd23a146784eabbd5e7a95d21e5d75 (127.1.46.193:33939)
I20260927 10:55:30.424575 10439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.193:60467
W20260927 10:55:30.529069 10630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:30.529275 10630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:30.529307 10630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:30.533172 10630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:30.533218 10630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:30.533288 10630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:30.537915 10630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:30.538836 10630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:30.540043 10630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:30.549515 10638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.549528 10636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.549515 10635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:30.549732 10630 server_base.cc:1034] running on GCE node
I20260927 10:55:30.550258 10630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:30.550853 10630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:30.552006 10630 hybrid_clock.cc:648] HybridClock initialized: now 1790506530551991 us; error 33 us; skew 500 ppm
I20260927 10:55:30.553680 10630 webserver.cc:460] Webserver started at http://127.1.46.194:35639/ using document root <none> and password file <none>
I20260927 10:55:30.554217 10630 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:30.554277 10630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:30.554481 10630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:30.556394 10630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "1b153e98b84a4973a80b6f3842ffbd03"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "bcae059cadfc242a8b88396d7330016a"
server_key_iv: "52855cbe16f6ff4e8a805838329c988b"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.556919 10630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "1b153e98b84a4973a80b6f3842ffbd03"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "bcae059cadfc242a8b88396d7330016a"
server_key_iv: "52855cbe16f6ff4e8a805838329c988b"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.560439 10630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:55:30.562924 10644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.564052 10630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:30.564184 10630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "1b153e98b84a4973a80b6f3842ffbd03"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "bcae059cadfc242a8b88396d7330016a"
server_key_iv: "52855cbe16f6ff4e8a805838329c988b"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.564291 10630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:30.589421 10630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:30.590049 10630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:30.590231 10630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:30.590839 10630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:30.591694 10630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:30.591771 10630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.591835 10630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:30.591876 10630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.601310 10630 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:44157
I20260927 10:55:30.601388 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:44157 every 8 connection(s)
I20260927 10:55:30.602212 10630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:30.611153 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:30.604024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:50855 (local address 127.1.46.254:39787)
0927 10:55:30.604168 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:30.604172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:30.605239 (+ 1067us) server_negotiation.cc:407] Connection header received
0927 10:55:30.606584 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:30.606588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:30.606638 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:30.606737 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:30.607596 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.608136 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.609304 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.609512 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.609612 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:30.610742 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:30.610745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:30.610747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:30.610759 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:30.610777 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:30.610785 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:30.610850 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:30.610985 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 10:55:30.611037 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:55:30.611922 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10630
I20260927 10:55:30.611987 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:30.604444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39787 (local address 127.1.46.194:50855)
0927 10:55:30.605083 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:30.605122 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:30.606355 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:30.606888 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:30.606894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:30.606971 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:30.607448 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:30.607459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.608310 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.608313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:30.609166 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:30.609175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.609698 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.609702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:30.609847 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:30.610421 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:55:30.610423 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:30.610459 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:55:30.610517 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:30.610864 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:30.610871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:30.610875 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:55:30.610969 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:55:30.611172 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260927 10:55:30.612051 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:30.612354 1211 external_mini_cluster.cc:1428] Setting key 96842fb687d60e00a1a21347591a2b40
I20260927 10:55:30.613201 10758 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39787
I20260927 10:55:30.613440 10758 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:30.614041 10758 heartbeater.cc:511] Master 127.1.46.254:39787 requested a full tablet report, sending...
I20260927 10:55:30.614393 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39787
--builtin_ntp_servers=127.1.46.212:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:55:30.615145 10439 ts_manager.cc:194] Registered new tserver with Master: 1b153e98b84a4973a80b6f3842ffbd03 (127.1.46.194:44157)
I20260927 10:55:30.615758 10439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.194:50855
W20260927 10:55:30.722725 10761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:30.722977 10761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:30.723043 10761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:30.726970 10761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:30.727034 10761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:30.727118 10761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:30.731623 10761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:30.732568 10761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:30.733826 10761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:30.742825 10767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.742846 10766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.742846 10769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:30.742893 10761 server_base.cc:1034] running on GCE node
I20260927 10:55:30.743489 10761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:30.744089 10761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:30.745271 10761 hybrid_clock.cc:648] HybridClock initialized: now 1790506530745257 us; error 29 us; skew 500 ppm
I20260927 10:55:30.746980 10761 webserver.cc:460] Webserver started at http://127.1.46.195:36249/ using document root <none> and password file <none>
I20260927 10:55:30.747511 10761 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:30.747596 10761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:30.747819 10761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:30.749786 10761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "56385e751b764f4b9803cc63e1441f0b"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "e06b1a85dca2777a1f25ea2d0edf6a5d"
server_key_iv: "790312830676b134d840dcccdab35b85"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.750312 10761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "56385e751b764f4b9803cc63e1441f0b"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "e06b1a85dca2777a1f25ea2d0edf6a5d"
server_key_iv: "790312830676b134d840dcccdab35b85"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.753690 10761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:55:30.756104 10775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.757237 10761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:55:30.757380 10761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "56385e751b764f4b9803cc63e1441f0b"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "e06b1a85dca2777a1f25ea2d0edf6a5d"
server_key_iv: "790312830676b134d840dcccdab35b85"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.757493 10761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:30.776296 10761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:30.776939 10761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:30.777141 10761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:30.777716 10761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:30.778638 10761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:30.778713 10761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.778790 10761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:30.778837 10761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.788168 10761 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:32851
I20260927 10:55:30.788205 10888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:32851 every 8 connection(s)
I20260927 10:55:30.789176 10761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:30.791141 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10761
I20260927 10:55:30.791255 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:30.791527 1211 external_mini_cluster.cc:1428] Setting key ca4130aff6885d50350fc00724f54077
I20260927 10:55:30.798350 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:30.791014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:34913 (local address 127.1.46.254:39787)
0927 10:55:30.791183 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:30.791190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:30.792448 (+ 1258us) server_negotiation.cc:407] Connection header received
0927 10:55:30.793970 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:30.793973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:30.794015 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:30.794091 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:30.795071 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.795606 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.796710 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.796923 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.796998 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:30.797965 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:30.797968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:30.797970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:30.797981 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:30.798000 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:30.798007 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:30.798064 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:30.798193 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 10:55:30.798240 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:55:30.799269 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:30.791493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39787 (local address 127.1.46.195:34913)
0927 10:55:30.792297 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:30.792352 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:30.793763 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:30.794284 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:30.794291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:30.794373 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:30.794935 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:30.794949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.795738 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.795741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:30.796610 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:30.796618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:30.797046 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:30.797049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:30.797187 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:30.797741 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:55:30.797742 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:30.797779 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:55:30.797838 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:30.798077 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:30.798083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:30.798087 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:55:30.798177 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:55:30.798438 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":182,"threads_started":1}
I20260927 10:55:30.800424 10889 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39787
I20260927 10:55:30.800647 10889 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:30.801196 10889 heartbeater.cc:511] Master 127.1.46.254:39787 requested a full tablet report, sending...
I20260927 10:55:30.802201 10439 ts_manager.cc:194] Registered new tserver with Master: 56385e751b764f4b9803cc63e1441f0b (127.1.46.195:32851)
I20260927 10:55:30.802827 10439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.195:34913
I20260927 10:55:30.805275 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:55:30.807816 10478 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:30.807098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.1.46.254:39787)
0927 10:55:30.807205 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:30.807209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:30.807381 (+ 172us) server_negotiation.cc:407] Connection header received
0927 10:55:30.807459 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:30.807461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:30.807496 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:30.807561 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:30.807733 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35684: BlockingRecv error: recv got EOF from 127.0.0.1:35684 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:55:30.808024 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10499
I20260927 10:55:30.813548 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10630
I20260927 10:55:30.818418 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10761
I20260927 10:55:30.823370 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10408
2026-09-27T10:55:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (855 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T10:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:30Z Disabled control of system clock
I20260927 10:55:30.851379 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:39059
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:39059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:55:30.958120 10900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:30.958356 10900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:30.958390 10900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:30.962317 10900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:30.962368 10900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:30.962383 10900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:30.962397 10900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:30.962412 10900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:30.967108 10900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:39059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:39059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:30.968050 10900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:30.969372 10900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:30.977756 10905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:30.977792 10906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:30.977838 10900 server_base.cc:1034] running on GCE node
W20260927 10:55:30.977792 10908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:30.978551 10900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:30.979744 10900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:30.980911 10900 hybrid_clock.cc:648] HybridClock initialized: now 1790506530980891 us; error 31 us; skew 500 ppm
I20260927 10:55:30.982517 10900 webserver.cc:460] Webserver started at http://127.1.46.254:41027/ using document root <none> and password file <none>
I20260927 10:55:30.983101 10900 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:30.983161 10900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:30.983312 10900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:30.985205 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "957a6c00cde04b4aaf3cd5414310d920"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "64f2b1f80c5c04d6c57b70120cd3f597"
server_key_iv: "85f5725b85c6842bb351a72043fdbf70"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.985760 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "957a6c00cde04b4aaf3cd5414310d920"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "64f2b1f80c5c04d6c57b70120cd3f597"
server_key_iv: "85f5725b85c6842bb351a72043fdbf70"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.989113 10900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:55:30.991380 10914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:30.992498 10900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:30.992619 10900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "957a6c00cde04b4aaf3cd5414310d920"
format_stamp: "Formatted at 2026-09-27 10:55:30 on dist-test-slave-cffh"
server_key: "64f2b1f80c5c04d6c57b70120cd3f597"
server_key_iv: "85f5725b85c6842bb351a72043fdbf70"
server_key_version: "encryptionkey@0"
I20260927 10:55:30.992735 10900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:30.997112 10900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:30.997654 10900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:30.997833 10900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:31.004236 10900 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:39059
I20260927 10:55:31.004262 10966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:39059 every 8 connection(s)
I20260927 10:55:31.005084 10900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:31.006006 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10900
I20260927 10:55:31.006093 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:55:31.006374 1211 external_mini_cluster.cc:1428] Setting key 4ed89bd226762efcef515a3826f9dfbd
I20260927 10:55:31.008097 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:31.012238 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:31.007729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.1.46.254:39059)
0927 10:55:31.008882 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:31.008897 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:31.008930 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:55:31.009541 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:31.009556 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:31.009883 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:31.010256 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:31.011050 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:31.011056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:31.011072 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:31.011090 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:31.011139 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:31.011154 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:31.011305 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:31.011375 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 10:55:31.011610 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":697,"threads_started":1}
I20260927 10:55:31.014225 10967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920: Bootstrap starting.
I20260927 10:55:31.016485 10967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:31.017238 10967 log.cc:824] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:31.019150 10967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920: No bootstrap required, opened a new log
I20260927 10:55:31.022711 10967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957a6c00cde04b4aaf3cd5414310d920" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39059 } }
I20260927 10:55:31.022917 10967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:31.022993 10967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a6c00cde04b4aaf3cd5414310d920, State: Initialized, Role: FOLLOWER
I20260927 10:55:31.023379 10967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957a6c00cde04b4aaf3cd5414310d920" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39059 } }
I20260927 10:55:31.023496 10967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:31.023584 10967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:31.023716 10967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:31.024544 10967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957a6c00cde04b4aaf3cd5414310d920" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39059 } }
I20260927 10:55:31.024899 10967 leader_election.cc:302] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 957a6c00cde04b4aaf3cd5414310d920; no voters:
I20260927 10:55:31.025211 10967 leader_election.cc:288] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:31.025349 10972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:31.025593 10972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [term 1 LEADER]: Becoming Leader. State: Replica: 957a6c00cde04b4aaf3cd5414310d920, State: Running, Role: LEADER
I20260927 10:55:31.025938 10972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957a6c00cde04b4aaf3cd5414310d920" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39059 } }
I20260927 10:55:31.026455 10967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:31.027493 10974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957a6c00cde04b4aaf3cd5414310d920. Latest consensus state: current_term: 1 leader_uuid: "957a6c00cde04b4aaf3cd5414310d920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957a6c00cde04b4aaf3cd5414310d920" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39059 } } }
I20260927 10:55:31.027580 10973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "957a6c00cde04b4aaf3cd5414310d920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957a6c00cde04b4aaf3cd5414310d920" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39059 } } }
I20260927 10:55:31.027627 10974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:31.027667 10973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:31.028008 10981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:31.031190 10981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:31.035989 10981 catalog_manager.cc:1347] Generated new cluster ID: 8b79712f28a14d6fbe567e0cc556cbc8
I20260927 10:55:31.036047 10981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:31.047199 10981 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:31.047466 10981 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:31.052199 10981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 957a6c00cde04b4aaf3cd5414310d920: Generated new TSK 0
I20260927 10:55:31.052762 10981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:55:31.055626 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39059
--builtin_ntp_servers=127.1.46.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:55:31.166353 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:31.166666 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:31.166740 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:31.170771 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:31.170826 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:31.170882 10991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:31.175676 10991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:31.176734 10991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:31.178093 10991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:31.187500 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:31.187559 10991 server_base.cc:1034] running on GCE node
W20260927 10:55:31.187440 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:31.187747 10999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:31.188154 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:31.188720 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:31.189888 10991 hybrid_clock.cc:648] HybridClock initialized: now 1790506531189861 us; error 39 us; skew 500 ppm
I20260927 10:55:31.191649 10991 webserver.cc:460] Webserver started at http://127.1.46.193:41839/ using document root <none> and password file <none>
I20260927 10:55:31.192209 10991 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:31.192271 10991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:31.192494 10991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:31.194492 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "95c680d94c78427bac53b6a08a983e2d"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "54ec782408c5ebe88bbba9f7348e9c2c"
server_key_iv: "8f703fbe24458761eb2e08865307f41e"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.195087 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "95c680d94c78427bac53b6a08a983e2d"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "54ec782408c5ebe88bbba9f7348e9c2c"
server_key_iv: "8f703fbe24458761eb2e08865307f41e"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.198563 10991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:55:31.200910 11005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.202101 10991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:31.202241 10991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "95c680d94c78427bac53b6a08a983e2d"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "54ec782408c5ebe88bbba9f7348e9c2c"
server_key_iv: "8f703fbe24458761eb2e08865307f41e"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.202350 10991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:31.206693 10991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:31.207262 10991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:31.207439 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:31.207952 10991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:31.208781 10991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:31.208849 10991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.208910 10991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:31.208964 10991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.217155 10991 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:32907
I20260927 10:55:31.217178 11118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:32907 every 8 connection(s)
I20260927 10:55:31.217948 10991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:31.222366 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 10991
I20260927 10:55:31.222472 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:31.222849 1211 external_mini_cluster.cc:1428] Setting key 7ec6520e22efc1c2a19183dd1ea4b606
I20260927 10:55:31.223935 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:31.219693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:41543 (local address 127.1.46.254:39059)
0927 10:55:31.219829 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:31.219833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:31.220779 (+ 946us) server_negotiation.cc:407] Connection header received
0927 10:55:31.222039 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:31.222041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:31.222081 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:31.222161 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:31.223458 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:31.223465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:31.223468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:31.223483 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:31.223508 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:31.223527 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:31.223598 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:31.223777 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 10:55:31.223817 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:55:31.224854 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:31.220020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39059 (local address 127.1.46.193:41543)
0927 10:55:31.220606 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:31.220644 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:31.221823 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:31.222362 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:31.222371 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:31.222486 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:31.223162 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:55:31.223164 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:31.223214 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:55:31.223303 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:31.223616 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:31.223626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:31.223634 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:55:31.223786 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 10:55:31.223958 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260927 10:55:31.224939 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39059
--builtin_ntp_servers=127.1.46.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 10:55:31.226104 11119 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39059
I20260927 10:55:31.226344 11119 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:31.226810 11119 heartbeater.cc:511] Master 127.1.46.254:39059 requested a full tablet report, sending...
I20260927 10:55:31.228361 10931 ts_manager.cc:194] Registered new tserver with Master: 95c680d94c78427bac53b6a08a983e2d (127.1.46.193:32907)
W20260927 10:55:31.331339 11122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:31.331555 11122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:31.331590 11122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:31.335340 11122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:31.335381 11122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:31.335431 11122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:31.339892 11122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:31.340817 11122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:31.342103 11122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:31.351370 11128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:31.351359 11127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:31.351362 11130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:31.351794 11122 server_base.cc:1034] running on GCE node
I20260927 10:55:31.352275 11122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:31.352902 11122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:31.354100 11122 hybrid_clock.cc:648] HybridClock initialized: now 1790506531354066 us; error 48 us; skew 500 ppm
I20260927 10:55:31.355865 11122 webserver.cc:460] Webserver started at http://127.1.46.194:35025/ using document root <none> and password file <none>
I20260927 10:55:31.356413 11122 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:31.356501 11122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:31.356721 11122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:31.358737 11122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "a2f1ac20613446d98285214b9f1a4a22"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "d0719ebfc2973a33ce38d683701d959e"
server_key_iv: "629298c19af374147f2f32088a46d184"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.359285 11122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "a2f1ac20613446d98285214b9f1a4a22"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "d0719ebfc2973a33ce38d683701d959e"
server_key_iv: "629298c19af374147f2f32088a46d184"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.362756 11122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:55:31.365036 11136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.366115 11122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:31.366240 11122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "a2f1ac20613446d98285214b9f1a4a22"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "d0719ebfc2973a33ce38d683701d959e"
server_key_iv: "629298c19af374147f2f32088a46d184"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.366345 11122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:31.370481 11122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:31.371042 11122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:31.371210 11122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:31.371688 11122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:31.372479 11122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:31.372545 11122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.372606 11122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:31.372655 11122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.381229 11122 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:44863
I20260927 10:55:31.381349 11249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:44863 every 8 connection(s)
I20260927 10:55:31.382027 11122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:31.387609 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:31.383782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:37723 (local address 127.1.46.254:39059)
0927 10:55:31.383929 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:31.383934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:31.384888 (+ 954us) server_negotiation.cc:407] Connection header received
0927 10:55:31.386037 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:31.386040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:31.386078 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:31.386154 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:31.387213 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:31.387218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:31.387221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:31.387234 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:31.387256 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:31.387265 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:31.387357 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:31.387451 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 10:55:31.387488 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:55:31.388392 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:31.384118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39059 (local address 127.1.46.194:37723)
0927 10:55:31.384739 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:31.384773 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:31.385854 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:31.386269 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:31.386274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:31.386381 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:31.386953 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:55:31.386954 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:31.386990 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:55:31.387066 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:31.387342 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:31.387346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:31.387349 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:55:31.387464 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:55:31.387687 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260927 10:55:31.389653 11250 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39059
I20260927 10:55:31.389878 11250 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:31.390301 11250 heartbeater.cc:511] Master 127.1.46.254:39059 requested a full tablet report, sending...
I20260927 10:55:31.391119 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 11122
I20260927 10:55:31.391248 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:31.391428 10931 ts_manager.cc:194] Registered new tserver with Master: a2f1ac20613446d98285214b9f1a4a22 (127.1.46.194:44863)
I20260927 10:55:31.391567 1211 external_mini_cluster.cc:1428] Setting key fa5bb495e8bd1019e412fca95a37bfb4
I20260927 10:55:31.393524 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39059
--builtin_ntp_servers=127.1.46.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:55:31.501395 11253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:31.501645 11253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:31.501719 11253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:31.505690 11253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:31.505759 11253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:31.505837 11253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:31.510411 11253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:31.511432 11253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:31.512715 11253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:31.522022 11261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:31.522079 11253 server_base.cc:1034] running on GCE node
W20260927 10:55:31.522029 11258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:31.522029 11259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:31.522712 11253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:31.523233 11253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:31.524396 11253 hybrid_clock.cc:648] HybridClock initialized: now 1790506531524377 us; error 31 us; skew 500 ppm
I20260927 10:55:31.525935 11253 webserver.cc:460] Webserver started at http://127.1.46.195:40189/ using document root <none> and password file <none>
I20260927 10:55:31.526502 11253 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:31.526625 11253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:31.526851 11253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:31.528929 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "b185ffed0a2d430187af00e96120abb1"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "be10073bee9136daf3e880703c2d0e09"
server_key_iv: "8c30690e27e2eae21a0287bfffb3cc76"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.529484 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "b185ffed0a2d430187af00e96120abb1"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "be10073bee9136daf3e880703c2d0e09"
server_key_iv: "8c30690e27e2eae21a0287bfffb3cc76"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.538532 11253 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260927 10:55:31.541062 11267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.542258 11253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:55:31.542410 11253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "b185ffed0a2d430187af00e96120abb1"
format_stamp: "Formatted at 2026-09-27 10:55:31 on dist-test-slave-cffh"
server_key: "be10073bee9136daf3e880703c2d0e09"
server_key_iv: "8c30690e27e2eae21a0287bfffb3cc76"
server_key_version: "encryptionkey@0"
I20260927 10:55:31.542546 11253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:31.546806 11253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:31.547456 11253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:31.547613 11253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:31.548079 11253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:31.548858 11253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:31.548902 11253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.548936 11253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:31.548952 11253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:31.557245 11253 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39307
I20260927 10:55:31.557314 11380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39307 every 8 connection(s)
I20260927 10:55:31.558058 11253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:31.559196 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 11253
I20260927 10:55:31.559314 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:31.559648 1211 external_mini_cluster.cc:1428] Setting key 943a2d11c4bb1cf0d9c2aa5a16072423
I20260927 10:55:31.564157 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:31.559903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:41217 (local address 127.1.46.254:39059)
0927 10:55:31.560031 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:31.560035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:31.561139 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 10:55:31.562474 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:31.562477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:31.562517 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:31.562636 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:31.563691 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:31.563695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:31.563698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:55:31.563711 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:31.563733 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:55:31.563755 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:31.563848 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:31.563995 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:55:31.564037 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:55:31.564966 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:31.560280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39059 (local address 127.1.46.195:41217)
0927 10:55:31.560992 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:31.561043 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:31.562273 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:31.562790 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:31.562798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:31.562883 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:31.563449 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:55:31.563451 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:31.563488 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:55:31.563550 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:31.563847 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:31.563856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:31.563863 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:55:31.564022 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 10:55:31.564236 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
I20260927 10:55:31.565989 11381 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39059
I20260927 10:55:31.566151 11381 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:31.566460 11381 heartbeater.cc:511] Master 127.1.46.254:39059 requested a full tablet report, sending...
I20260927 10:55:31.567440 10931 ts_manager.cc:194] Registered new tserver with Master: b185ffed0a2d430187af00e96120abb1 (127.1.46.195:39307)
I20260927 10:55:31.573271 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:55:31.575855 10970 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:55:31.575010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.1.46.254:39059)
0927 10:55:31.575153 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:31.575157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:31.575331 (+ 174us) server_negotiation.cc:407] Connection header received
0927 10:55:31.575419 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:31.575421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:31.575448 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:31.575503 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:31.575717 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50220: BlockingRecv error: recv got EOF from 127.0.0.1:50220 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:55:31.575981 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10991
I20260927 10:55:31.581300 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 11122
I20260927 10:55:31.586359 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 11253
I20260927 10:55:31.591382 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 10900
2026-09-27T10:55:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (768 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:31 dist-test-slave-cffh krb5kdc[11391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:31 dist-test-slave-cffh krb5kdc[11391](info): set up 2 sockets
Sep 27 10:55:31 dist-test-slave-cffh krb5kdc[11391](info): commencing operation
krb5kdc: starting...
W20260927 10:55:32.619649 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:55:32 dist-test-slave-cffh krb5kdc[11391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:55:32.735437 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36089
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:36089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:32.840553 11404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:32.840773 11404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:32.840811 11404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:32.845530 11404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:55:32.845589 11404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:32.845650 11404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:32.845687 11404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:55:32.845718 11404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:55:32.850925 11404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:36089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:32.851809 11404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:32.852741 11404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:32.860839 11409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:32.860800 11410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:32.860864 11404 server_base.cc:1034] running on GCE node
W20260927 10:55:32.860800 11412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:32.861573 11404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:32.862528 11404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:32.863729 11404 hybrid_clock.cc:648] HybridClock initialized: now 1790506532863707 us; error 42 us; skew 500 ppm
Sep 27 10:55:32 dist-test-slave-cffh krb5kdc[11391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506532, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:32.866379 11404 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:55:32.867272 11404 webserver.cc:460] Webserver started at http://127.1.46.254:38399/ using document root <none> and password file <none>
I20260927 10:55:32.867789 11404 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:32.867880 11404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:32.868096 11404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:32.869982 11404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "bc1523f37304476580c843eee4adf591"
format_stamp: "Formatted at 2026-09-27 10:55:32 on dist-test-slave-cffh"
server_key: "e34e15de02803370192817b27fa8b374"
server_key_iv: "ec0e1886f5f57436642ea32638e10481"
server_key_version: "encryptionkey@0"
I20260927 10:55:32.870563 11404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "bc1523f37304476580c843eee4adf591"
format_stamp: "Formatted at 2026-09-27 10:55:32 on dist-test-slave-cffh"
server_key: "e34e15de02803370192817b27fa8b374"
server_key_iv: "ec0e1886f5f57436642ea32638e10481"
server_key_version: "encryptionkey@0"
I20260927 10:55:32.873960 11404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:55:32.876327 11419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:32.877398 11404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:55:32.877527 11404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "bc1523f37304476580c843eee4adf591"
format_stamp: "Formatted at 2026-09-27 10:55:32 on dist-test-slave-cffh"
server_key: "e34e15de02803370192817b27fa8b374"
server_key_iv: "ec0e1886f5f57436642ea32638e10481"
server_key_version: "encryptionkey@0"
I20260927 10:55:32.877629 11404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:32.887255 11404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:32.889931 11404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:32.890108 11404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:32.897073 11404 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:36089
I20260927 10:55:32.897362 11471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:36089 every 8 connection(s)
I20260927 10:55:32.897966 11404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:55:32.900533 11472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:32.900547 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 11404
I20260927 10:55:32.900664 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:55:32.900941 1211 external_mini_cluster.cc:1428] Setting key c9643ff428aa195a33023d985582995e
I20260927 10:55:32.907140 11472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591: Bootstrap starting.
Sep 27 10:55:32 dist-test-slave-cffh krb5kdc[11391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:32.909790 11472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:32.910673 11472 log.cc:824] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:32.912784 11472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591: No bootstrap required, opened a new log
I20260927 10:55:32.915205 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:32.902151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.1.46.254:36089)
0927 10:55:32.902529 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:32.902542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:32.902626 (+ 84us) server_negotiation.cc:407] Connection header received
0927 10:55:32.903432 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:32.903450 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:32.903742 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:32.904000 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:32.904906 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:32.905844 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:32.907112 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:32.907632 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:32.907768 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:32.910427 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:32.910432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:32.910443 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:32.910467 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:32.912477 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:32.913002 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:32.913006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:32.913010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:32.913111 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:32.913420 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:32.913423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:32.913424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:32.913752 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:32.913900 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:32.914203 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 10:55:32.914483 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260927 10:55:32.916631 11472 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1523f37304476580c843eee4adf591" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36089 } }
I20260927 10:55:32.916877 11472 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:32.916958 11472 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1523f37304476580c843eee4adf591, State: Initialized, Role: FOLLOWER
I20260927 10:55:32.917404 11472 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1523f37304476580c843eee4adf591" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36089 } }
I20260927 10:55:32.917533 11472 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:55:32.917584 11472 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:55:32.917656 11472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:32.918462 11472 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1523f37304476580c843eee4adf591" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36089 } }
I20260927 10:55:32.918808 11472 leader_election.cc:302] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc1523f37304476580c843eee4adf591; no voters:
I20260927 10:55:32.919050 11472 leader_election.cc:288] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:55:32.919173 11477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:32.919410 11477 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [term 1 LEADER]: Becoming Leader. State: Replica: bc1523f37304476580c843eee4adf591, State: Running, Role: LEADER
I20260927 10:55:32.919684 11477 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1523f37304476580c843eee4adf591" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36089 } }
I20260927 10:55:32.920212 11472 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:55:32.921375 11478 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc1523f37304476580c843eee4adf591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1523f37304476580c843eee4adf591" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36089 } } }
I20260927 10:55:32.921519 11478 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [sys.catalog]: This master's current role is: LEADER
I20260927 10:55:32.921384 11479 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc1523f37304476580c843eee4adf591. Latest consensus state: current_term: 1 leader_uuid: "bc1523f37304476580c843eee4adf591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1523f37304476580c843eee4adf591" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36089 } } }
I20260927 10:55:32.921770 11479 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591 [sys.catalog]: This master's current role is: LEADER
W20260927 10:55:32.924615 11493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:55:32.924678 11493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:55:32.924754 11486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:55:32.925414 11486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:55:32.930096 11486 catalog_manager.cc:1347] Generated new cluster ID: 68d5ea905b31461fbd172c7011a8ad30
I20260927 10:55:32.930153 11486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:55:32.941462 11486 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:55:32.942791 11486 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:55:32.957702 11486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591: Generated new TSK 0
I20260927 10:55:32.958557 11486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:55:32.985445 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36089
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:33.093645 11498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:33.093894 11498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:33.093936 11498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:33.097986 11498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:33.098047 11498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:33.098104 11498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:55:33.103413 11498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:33.104411 11498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:33.105413 11498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:33.114784 11506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:33.114794 11503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:33.114794 11504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:33.114828 11498 server_base.cc:1034] running on GCE node
I20260927 10:55:33.115437 11498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:33.115998 11498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:33.117201 11498 hybrid_clock.cc:648] HybridClock initialized: now 1790506533117186 us; error 38 us; skew 500 ppm
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:33.120169 11498 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:55:33.121246 11498 webserver.cc:460] Webserver started at http://127.1.46.193:42461/ using document root <none> and password file <none>
I20260927 10:55:33.121893 11498 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:33.121989 11498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:33.122268 11498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:33.124348 11498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "dbd4cfbea51e4386acca367f4bfe299c"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "63e5bc929c8d22cd96d423be7ad6328d"
server_key_iv: "fe1a2e6b117c4d1358a67ea1705daa91"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.124915 11498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "dbd4cfbea51e4386acca367f4bfe299c"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "63e5bc929c8d22cd96d423be7ad6328d"
server_key_iv: "fe1a2e6b117c4d1358a67ea1705daa91"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.128739 11498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:55:33.131131 11513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.132347 11498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:33.132491 11498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "dbd4cfbea51e4386acca367f4bfe299c"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "63e5bc929c8d22cd96d423be7ad6328d"
server_key_iv: "fe1a2e6b117c4d1358a67ea1705daa91"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.132597 11498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:33.153118 11498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:33.156306 11498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:33.156495 11498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:33.157047 11498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:33.157820 11498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:33.157886 11498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.157953 11498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:33.157995 11498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.166224 11498 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:38075
I20260927 10:55:33.166247 11626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:38075 every 8 connection(s)
I20260927 10:55:33.167085 11498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:55:33.170838 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 11498
I20260927 10:55:33.170944 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:55:33.171262 1211 external_mini_cluster.cc:1428] Setting key 49cf96b8b6a708e7bcfe099450fc18a7
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:55:33.181226 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.168825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:38159 (local address 127.1.46.254:36089)
0927 10:55:33.168961 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.168968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.169737 (+ 769us) server_negotiation.cc:407] Connection header received
0927 10:55:33.171380 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.171382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.171428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.171499 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:33.172792 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.173313 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.174893 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.175130 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.175241 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.177645 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:33.177648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:33.177650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:33.177673 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:33.179082 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.179583 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.179588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.179591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.179669 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.180004 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.180010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.180012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.180185 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:33.180288 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.180880 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.181042 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:55:33.182142 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.169178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36089 (local address 127.1.46.193:38159)
0927 10:55:33.169600 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.169638 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.171130 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.171680 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.171687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.172079 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:33.172633 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.172647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.173438 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.173444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.174770 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.174779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.175271 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.175275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.175473 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.176080 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:33.176095 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:33.177478 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:33.179210 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:33.179217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:33.179230 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:33.179482 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:33.179770 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:33.179774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:33.179777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:33.179911 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:33.180296 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:33.180305 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:33.180664 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.180848 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.181152 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:55:33.183343 11627 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36089
I20260927 10:55:33.183597 11627 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:33.184154 11627 heartbeater.cc:511] Master 127.1.46.254:36089 requested a full tablet report, sending...
I20260927 10:55:33.185837 11436 ts_manager.cc:194] Registered new tserver with Master: dbd4cfbea51e4386acca367f4bfe299c (127.1.46.193:38075)
I20260927 10:55:33.187568 11436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:38159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:55:33.197289 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36089
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:33.302203 11632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:33.302466 11632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:33.302506 11632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:33.306473 11632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:33.306545 11632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:33.306681 11632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:55:33.311694 11632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:33.312631 11632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:33.313505 11632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:33.322646 11640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:33.322760 11632 server_base.cc:1034] running on GCE node
W20260927 10:55:33.322680 11637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:33.322678 11638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:33.323313 11632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:33.323843 11632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:33.324998 11632 hybrid_clock.cc:648] HybridClock initialized: now 1790506533324972 us; error 41 us; skew 500 ppm
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:33.327526 11632 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:55:33.328306 11632 webserver.cc:460] Webserver started at http://127.1.46.194:41493/ using document root <none> and password file <none>
I20260927 10:55:33.328822 11632 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:33.328917 11632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:33.329133 11632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:33.331126 11632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "b1ac7680c256435aab8460bfa89f4e2a"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "d066d5be7bbb1a9d310d21136725dc79"
server_key_iv: "15d62cdc6a7f7f1707eb23df1d97d548"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.331683 11632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "b1ac7680c256435aab8460bfa89f4e2a"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "d066d5be7bbb1a9d310d21136725dc79"
server_key_iv: "15d62cdc6a7f7f1707eb23df1d97d548"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.335073 11632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:55:33.337493 11647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.338833 11632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:33.338961 11632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "b1ac7680c256435aab8460bfa89f4e2a"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "d066d5be7bbb1a9d310d21136725dc79"
server_key_iv: "15d62cdc6a7f7f1707eb23df1d97d548"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.339066 11632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:33.352639 11632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:33.355644 11632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:33.355831 11632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:33.356354 11632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:33.357116 11632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:33.357183 11632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.357254 11632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:33.357306 11632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.365922 11632 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:43533
I20260927 10:55:33.365964 11760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:43533 every 8 connection(s)
I20260927 10:55:33.366968 11632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:55:33.372512 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 11632
I20260927 10:55:33.372639 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:55:33.372947 1211 external_mini_cluster.cc:1428] Setting key fa4cff94519130b71b270b394d0ff653
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:33.380776 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.368714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:60641 (local address 127.1.46.254:36089)
0927 10:55:33.368845 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.368857 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.369543 (+ 686us) server_negotiation.cc:407] Connection header received
0927 10:55:33.370923 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.370926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.370982 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.371055 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:33.372065 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.372628 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.373867 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.374215 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.374325 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.377216 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:33.377219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:33.377221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:33.377242 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:33.378701 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.379267 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.379270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.379271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.379340 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.379715 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.379717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.379719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.379897 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:33.379971 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.380582 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.380669 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:55:33.381841 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.368994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36089 (local address 127.1.46.194:60641)
0927 10:55:33.369408 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.369446 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.370708 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.371198 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.371204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.371513 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:33.371926 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.371937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.372773 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.372778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.373756 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.373764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.374361 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.374365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.374555 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.375145 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:33.375170 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:33.377057 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:33.378809 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:33.378816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:33.378829 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:33.379174 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:33.379431 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:33.379437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:33.379442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:33.379624 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:33.380002 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:33.380010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:33.380338 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.380545 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.380855 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:55:33.383038 11761 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36089
I20260927 10:55:33.383247 11761 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:33.383678 11761 heartbeater.cc:511] Master 127.1.46.254:36089 requested a full tablet report, sending...
I20260927 10:55:33.384696 11436 ts_manager.cc:194] Registered new tserver with Master: b1ac7680c256435aab8460bfa89f4e2a (127.1.46.194:43533)
I20260927 10:55:33.385272 11436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:60641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:55:33.401005 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36089
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:55:33.508342 11766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:55:33.508553 11766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:55:33.508591 11766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:55:33.512506 11766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:55:33.512552 11766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:55:33.512605 11766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:55:33.517513 11766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:55:33.518323 11766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:55:33.519282 11766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:55:33.528460 11774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:33.528491 11766 server_base.cc:1034] running on GCE node
W20260927 10:55:33.528491 11772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:55:33.528522 11771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:55:33.529095 11766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:55:33.529642 11766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:55:33.530820 11766 hybrid_clock.cc:648] HybridClock initialized: now 1790506533530808 us; error 34 us; skew 500 ppm
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:55:33.533335 11766 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:55:33.534085 11766 webserver.cc:460] Webserver started at http://127.1.46.195:38037/ using document root <none> and password file <none>
I20260927 10:55:33.534643 11766 fs_manager.cc:359] Metadata directory not provided
I20260927 10:55:33.534749 11766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:55:33.534976 11766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:55:33.537111 11766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "465beb83383d4e8fb9813800a1a5f978"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "79135e06011e3727764a28f5d3c1e7e0"
server_key_iv: "4beef5a661a096ffc8aa90c6dc4c408c"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.537652 11766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "465beb83383d4e8fb9813800a1a5f978"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "79135e06011e3727764a28f5d3c1e7e0"
server_key_iv: "4beef5a661a096ffc8aa90c6dc4c408c"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.541105 11766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:55:33.543424 11781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.544566 11766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:55:33.544692 11766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "465beb83383d4e8fb9813800a1a5f978"
format_stamp: "Formatted at 2026-09-27 10:55:33 on dist-test-slave-cffh"
server_key: "79135e06011e3727764a28f5d3c1e7e0"
server_key_iv: "4beef5a661a096ffc8aa90c6dc4c408c"
server_key_version: "encryptionkey@0"
I20260927 10:55:33.544836 11766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:55:33.556716 11766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:55:33.559795 11766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:55:33.559998 11766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:55:33.560498 11766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:55:33.561221 11766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:55:33.561290 11766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.561343 11766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:55:33.561381 11766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:55:33.569880 11766 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39807
I20260927 10:55:33.569905 11894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39807 every 8 connection(s)
I20260927 10:55:33.570704 11766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:55:33.576524 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 11766
I20260927 10:55:33.576642 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:55:33.576927 1211 external_mini_cluster.cc:1428] Setting key 5339742c2b341d0d5c6002dff9ebcdca
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506533, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:55:33.583951 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.572405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:50071 (local address 127.1.46.254:36089)
0927 10:55:33.572540 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.572544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.573268 (+ 724us) server_negotiation.cc:407] Connection header received
0927 10:55:33.574648 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.574652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.574707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.574815 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:33.575913 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.576432 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.577922 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.578233 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.578342 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.580797 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:33.580799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:33.580801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:33.580822 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:33.582112 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.582527 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.582530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.582531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.582670 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.583022 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.583024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.583025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.583184 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:33.583257 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.583752 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.583836 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:55:33.584715 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.572736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36089 (local address 127.1.46.195:50071)
0927 10:55:33.573126 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.573163 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.574403 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.574979 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.574986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.575313 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:55:33.575766 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.575778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.576610 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.576615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.577789 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.577802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.578409 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.578413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.578622 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.579242 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:55:33.579258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:55:33.580662 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:55:33.582204 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:33.582208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:33.582218 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:33.582446 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:33.582783 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:55:33.582787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:55:33.582789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:55:33.582917 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:55:33.583330 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:55:33.583335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:55:33.583560 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.583710 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.583942 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260927 10:55:33.585813 11895 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36089
I20260927 10:55:33.586063 11895 heartbeater.cc:464] Registering TS with master...
I20260927 10:55:33.586560 11895 heartbeater.cc:511] Master 127.1.46.254:36089 requested a full tablet report, sending...
I20260927 10:55:33.587620 11436 ts_manager.cc:194] Registered new tserver with Master: 465beb83383d4e8fb9813800a1a5f978 (127.1.46.195:39807)
I20260927 10:55:33.588218 11436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:50071
I20260927 10:55:33.591137 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:55:33.601190 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.593089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.1.46.254:36089)
0927 10:55:33.593251 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.593256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.593445 (+ 189us) server_negotiation.cc:407] Connection header received
0927 10:55:33.593607 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.593611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.593655 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.593754 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:55:33.594529 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.595129 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.596042 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.596295 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.596419 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.598024 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:55:33.598029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:55:33.598033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:55:33.598060 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:55:33.599395 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.599851 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.599855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.599858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.599943 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:55:33.600246 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:55:33.600250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:55:33.600253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:55:33.600474 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:55:33.600622 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.600887 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.601022 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:55:33.603905 11436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:55:33.607102 11436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:55:33.620957 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.615093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.1.46.193:38075)
0927 10:55:33.615418 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.615431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.615469 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:55:33.615735 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.615741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.615961 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.616122 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:33.616706 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.617765 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.619429 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.620264 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.620391 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.620534 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.620703 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.620795 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":149,"threads_started":1}
I20260927 10:55:33.620959 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.614919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:38075 (local address 127.0.0.1:49218)
0927 10:55:33.615346 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.615403 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.615528 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.616393 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.616399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.616425 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:33.616568 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.616575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.617927 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.617931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.619290 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.619302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.620395 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.620398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.620503 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.620538 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.620649 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.620798 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":65,"threads_started":1}
I20260927 10:55:33.621421 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.615578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39807 (local address 127.0.0.1:58992)
0927 10:55:33.616025 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.616038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.616127 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.616843 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.616846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.616870 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:33.616988 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.616994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.618346 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.618350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.619654 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.619661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.621002 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.621007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.621089 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.621112 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.621196 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.621264 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"spinlock_wait_cycles":3328,"thread_start_us":92,"threads_started":1}
I20260927 10:55:33.621555 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.615783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.1.46.195:39807)
0927 10:55:33.616218 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.616229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.616262 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:55:33.616333 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.616338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.616518 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.616664 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:33.617106 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.618171 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.619826 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.620676 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.620802 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.620942 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.621230 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.621333 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":136,"threads_started":1}
I20260927 10:55:33.623731 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.615893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:43533 (local address 127.0.0.1:36722)
0927 10:55:33.616740 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.616753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.616868 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.618459 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.618463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.618480 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:33.618599 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.618605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.619993 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.619997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.622192 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.622205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.623333 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.623336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.623427 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.623450 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.623546 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.623610 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"spinlock_wait_cycles":1920,"thread_start_us":82,"threads_started":1}
I20260927 10:55:33.623860 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.616372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.1.46.194:43533)
0927 10:55:33.618048 (+ 1676us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.618058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.618076 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:55:33.618138 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.618142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.618326 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.618478 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:33.618721 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.619866 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.622365 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.623193 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.623331 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.623470 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.623605 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.623707 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1581,"thread_start_us":145,"threads_started":1}
I20260927 10:55:33.624274 11561 tablet_service.cc:1467] Processing CreateTablet for tablet 02bca41028934eabbebf993000b0c5cf (DEFAULT_TABLE table=test-table [id=6663fb4d9e144fe88639cd0d24c729e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:33.624274 11829 tablet_service.cc:1467] Processing CreateTablet for tablet 02bca41028934eabbebf993000b0c5cf (DEFAULT_TABLE table=test-table [id=6663fb4d9e144fe88639cd0d24c729e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:33.625294 11561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bca41028934eabbebf993000b0c5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:33.625362 11829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bca41028934eabbebf993000b0c5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:33.626636 11695 tablet_service.cc:1467] Processing CreateTablet for tablet 02bca41028934eabbebf993000b0c5cf (DEFAULT_TABLE table=test-table [id=6663fb4d9e144fe88639cd0d24c729e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:55:33.627597 11695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bca41028934eabbebf993000b0c5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:55:33.631270 11912 tablet_bootstrap.cc:492] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: Bootstrap starting.
I20260927 10:55:33.633134 11913 tablet_bootstrap.cc:492] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Bootstrap starting.
I20260927 10:55:33.633649 11912 tablet_bootstrap.cc:654] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:33.634375 11914 tablet_bootstrap.cc:492] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Bootstrap starting.
I20260927 10:55:33.634948 11912 log.cc:824] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:33.635370 11913 tablet_bootstrap.cc:654] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:33.636554 11913 log.cc:824] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:33.636933 11914 tablet_bootstrap.cc:654] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Neither blocks nor log segments found. Creating new log.
I20260927 10:55:33.637182 11912 tablet_bootstrap.cc:492] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: No bootstrap required, opened a new log
I20260927 10:55:33.637375 11912 ts_tablet_manager.cc:1397] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:33.637740 11914 log.cc:824] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:55:33.638609 11913 tablet_bootstrap.cc:492] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: No bootstrap required, opened a new log
I20260927 10:55:33.638803 11913 ts_tablet_manager.cc:1397] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:33.640127 11914 tablet_bootstrap.cc:492] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: No bootstrap required, opened a new log
I20260927 10:55:33.640313 11914 ts_tablet_manager.cc:1397] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:33.640595 11912 raft_consensus.cc:348] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.640776 11912 raft_consensus.cc:374] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:33.640811 11912 raft_consensus.cc:729] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd4cfbea51e4386acca367f4bfe299c, State: Initialized, Role: FOLLOWER
I20260927 10:55:33.641186 11912 consensus_queue.cc:260] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.641749 11627 heartbeater.cc:503] Master 127.1.46.254:36089 was elected leader, sending a full tablet report...
I20260927 10:55:33.642138 11912 ts_tablet_manager.cc:1428] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:55:33.642722 11913 raft_consensus.cc:348] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.642992 11913 raft_consensus.cc:374] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:33.643095 11913 raft_consensus.cc:729] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465beb83383d4e8fb9813800a1a5f978, State: Initialized, Role: FOLLOWER
I20260927 10:55:33.643534 11913 consensus_queue.cc:260] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.644238 11895 heartbeater.cc:503] Master 127.1.46.254:36089 was elected leader, sending a full tablet report...
I20260927 10:55:33.644512 11913 ts_tablet_manager.cc:1428] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:55:33.645179 11914 raft_consensus.cc:348] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.645438 11914 raft_consensus.cc:374] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:55:33.645512 11914 raft_consensus.cc:729] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ac7680c256435aab8460bfa89f4e2a, State: Initialized, Role: FOLLOWER
I20260927 10:55:33.645951 11914 consensus_queue.cc:260] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.646625 11761 heartbeater.cc:503] Master 127.1.46.254:36089 was elected leader, sending a full tablet report...
I20260927 10:55:33.646953 11914 ts_tablet_manager.cc:1428] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 10:55:33.668149 11628 tablet.cc:2369] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:33.789274 11919 raft_consensus.cc:482] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:55:33.789520 11919 raft_consensus.cc:504] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.790845 11919 leader_election.cc:288] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd4cfbea51e4386acca367f4bfe299c (127.1.46.193:38075), b1ac7680c256435aab8460bfa89f4e2a (127.1.46.194:43533)
I20260927 10:55:33.795044 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.791106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:38075 (local address 127.1.46.195:51971)
0927 10:55:33.791254 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.791274 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.791416 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.791748 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.791753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.791777 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:33.791948 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.791953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.793005 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.793008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.793829 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.793837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.794598 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.794605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.794709 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.794738 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.794820 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.794887 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 10:55:33.795070 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.791188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:51971 (local address 127.1.46.193:38075)
0927 10:55:33.791342 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.791346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.791365 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:55:33.791562 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.791567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.791627 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.791738 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:33.792084 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.792853 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.793991 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.794450 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.794549 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.794659 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.794849 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.794939 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:55:33.795140 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.791525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:34391 (local address 127.1.46.194:43533)
0927 10:55:33.791643 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.791647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.791881 (+ 234us) server_negotiation.cc:407] Connection header received
0927 10:55:33.792026 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.792031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.792120 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.792179 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:55:33.792482 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.793020 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.794105 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.794524 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.794666 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.794717 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.794950 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.795014 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:55:33.795190 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.791400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:43533 (local address 127.1.46.195:34391)
0927 10:55:33.791782 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.791797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:55:33.791916 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:55:33.792216 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:55:33.792219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:55:33.792238 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:55:33.792365 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.792374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.793114 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.793116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.793979 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:55:33.793989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.794659 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.794664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:55:33.794809 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.794839 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:55:33.794924 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:55:33.794991 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260927 10:55:33.795928 11581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bca41028934eabbebf993000b0c5cf" candidate_uuid: "465beb83383d4e8fb9813800a1a5f978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd4cfbea51e4386acca367f4bfe299c" is_pre_election: true
I20260927 10:55:33.795925 11715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bca41028934eabbebf993000b0c5cf" candidate_uuid: "465beb83383d4e8fb9813800a1a5f978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac7680c256435aab8460bfa89f4e2a" is_pre_election: true
I20260927 10:55:33.796214 11715 raft_consensus.cc:2454] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465beb83383d4e8fb9813800a1a5f978 in term 0.
I20260927 10:55:33.796211 11581 raft_consensus.cc:2454] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465beb83383d4e8fb9813800a1a5f978 in term 0.
I20260927 10:55:33.796695 11783 leader_election.cc:302] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465beb83383d4e8fb9813800a1a5f978, dbd4cfbea51e4386acca367f4bfe299c; no voters:
I20260927 10:55:33.796979 11919 raft_consensus.cc:2781] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:55:33.797071 11919 raft_consensus.cc:482] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:55:33.797120 11919 raft_consensus.cc:3037] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:33.798184 11919 raft_consensus.cc:504] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.798527 11919 leader_election.cc:288] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [CANDIDATE]: Term 1 election: Requested vote from peers dbd4cfbea51e4386acca367f4bfe299c (127.1.46.193:38075), b1ac7680c256435aab8460bfa89f4e2a (127.1.46.194:43533)
I20260927 10:55:33.798924 11581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bca41028934eabbebf993000b0c5cf" candidate_uuid: "465beb83383d4e8fb9813800a1a5f978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd4cfbea51e4386acca367f4bfe299c"
I20260927 10:55:33.798959 11715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bca41028934eabbebf993000b0c5cf" candidate_uuid: "465beb83383d4e8fb9813800a1a5f978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac7680c256435aab8460bfa89f4e2a"
I20260927 10:55:33.799065 11715 raft_consensus.cc:3037] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:33.799058 11581 raft_consensus.cc:3037] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:55:33.800369 11581 raft_consensus.cc:2454] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465beb83383d4e8fb9813800a1a5f978 in term 1.
I20260927 10:55:33.800369 11715 raft_consensus.cc:2454] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465beb83383d4e8fb9813800a1a5f978 in term 1.
I20260927 10:55:33.800652 11783 leader_election.cc:302] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465beb83383d4e8fb9813800a1a5f978, b1ac7680c256435aab8460bfa89f4e2a; no voters:
I20260927 10:55:33.800940 11919 raft_consensus.cc:2781] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:55:33.801198 11919 raft_consensus.cc:686] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 1 LEADER]: Becoming Leader. State: Replica: 465beb83383d4e8fb9813800a1a5f978, State: Running, Role: LEADER
I20260927 10:55:33.801522 11919 consensus_queue.cc:237] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } }
I20260927 10:55:33.804719 11436 catalog_manager.cc:5495] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 reported cstate change: term changed from 0 to 1, leader changed from <none> to 465beb83383d4e8fb9813800a1a5f978 (127.1.46.195). New cstate: current_term: 1 leader_uuid: "465beb83383d4e8fb9813800a1a5f978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "465beb83383d4e8fb9813800a1a5f978" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 } health_report { overall_health: UNKNOWN } } }
W20260927 10:55:33.821687 11896 tablet.cc:2369] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:33.854892 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.851260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.1.46.195:39807)
0927 10:55:33.851389 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:55:33.851393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:55:33.851438 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:55:33.851601 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:55:33.851604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:55:33.851659 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:55:33.851748 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:55:33.851993 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.852472 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.853327 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:55:33.853529 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:55:33.853602 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:55:33.853927 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:55:33.854330 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:55:33.854422 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:55:33.854691 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 10:55:33.854752 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:55:33.860771 11581 raft_consensus.cc:1261] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 1 FOLLOWER]: Refusing update from remote peer 465beb83383d4e8fb9813800a1a5f978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:33.860771 11715 raft_consensus.cc:1261] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 1 FOLLOWER]: Refusing update from remote peer 465beb83383d4e8fb9813800a1a5f978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:55:33.861503 11919 consensus_queue.cc:1035] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ac7680c256435aab8460bfa89f4e2a" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:55:33.861703 11922 consensus_queue.cc:1035] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd4cfbea51e4386acca367f4bfe299c" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:55:33.868184 11762 tablet.cc:2369] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:55:33.869614 11925 mvcc.cc:205] Tried to move back new op lower bound from 7333914762686537728 to 7333914762453565440. Current Snapshot: MvccSnapshot[applied={T|T < 7333914762686537728}]
I20260927 10:55:33.871701 11926 mvcc.cc:205] Tried to move back new op lower bound from 7333914762686537728 to 7333914762453565440. Current Snapshot: MvccSnapshot[applied={T|T < 7333914762686537728 or (T in {7333914762686537728})}]
I20260927 10:55:33.880160 11435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:33.880321 11435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:55:33.883057 11435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc1523f37304476580c843eee4adf591: Sending DeleteTablet for 3 replicas of tablet 02bca41028934eabbebf993000b0c5cf
I20260927 10:55:33.883780 11561 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bca41028934eabbebf993000b0c5cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:33 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:49218
I20260927 10:55:33.884027 11829 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bca41028934eabbebf993000b0c5cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:33 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:58992
I20260927 10:55:33.884330 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 11498
I20260927 10:55:33.884397 11935 tablet_replica.cc:330] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c: stopping tablet replica
I20260927 10:55:33.884421 11695 tablet_service.cc:1514] Processing DeleteTablet for tablet 02bca41028934eabbebf993000b0c5cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:55:33 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:36722
I20260927 10:55:33.884706 11935 raft_consensus.cc:2229] T 02bca41028934eabbebf993000b0c5cf P dbd4cfbea51e4386acca367f4bfe299c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:33.885288 11937 tablet_replica.cc:330] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: stopping tablet replica
I20260927 10:55:33.885557 11937 raft_consensus.cc:2229] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:55:33.885839 11937 raft_consensus.cc:2258] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:33.887346 11937 ts_tablet_manager.cc:1905] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:55:33.890091 11937 ts_tablet_manager.cc:1918] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:33.890190 11937 log.cc:1196] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/02bca41028934eabbebf993000b0c5cf
I20260927 10:55:33.890458 11936 tablet_replica.cc:330] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: stopping tablet replica
I20260927 10:55:33.890549 11937 ts_tablet_manager.cc:1939] T 02bca41028934eabbebf993000b0c5cf P b1ac7680c256435aab8460bfa89f4e2a: Deleting consensus metadata
I20260927 10:55:33.890853 11936 raft_consensus.cc:2229] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:55:33.891315 11936 raft_consensus.cc:2258] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:55:33.892179 11421 catalog_manager.cc:4841] TS b1ac7680c256435aab8460bfa89f4e2a (127.1.46.194:43533): tablet 02bca41028934eabbebf993000b0c5cf (table test-table [id=6663fb4d9e144fe88639cd0d24c729e7]) successfully deleted
I20260927 10:55:33.892661 11936 ts_tablet_manager.cc:1905] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:55:33.893103 11421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:38075 (error 108)
I20260927 10:55:33.893247 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 11632
I20260927 10:55:33.895361 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:55:33.895047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:38075 (local address 127.0.0.1:49228)
0927 10:55:33.895193 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:55:33.895261 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 10:55:33.895493 11421 catalog_manager.cc:4584] TS dbd4cfbea51e4386acca367f4bfe299c (127.1.46.193:38075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02bca41028934eabbebf993000b0c5cf: Network error: Client connection negotiation failed: client connection to 127.1.46.193:38075: connect: Connection refused (error 111)
I20260927 10:55:33.895983 11936 ts_tablet_manager.cc:1918] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:55:33.896090 11936 log.cc:1196] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/02bca41028934eabbebf993000b0c5cf
I20260927 10:55:33.896387 11936 ts_tablet_manager.cc:1939] T 02bca41028934eabbebf993000b0c5cf P 465beb83383d4e8fb9813800a1a5f978: Deleting consensus metadata
I20260927 10:55:33.897437 11421 catalog_manager.cc:4841] TS 465beb83383d4e8fb9813800a1a5f978 (127.1.46.195:39807): tablet 02bca41028934eabbebf993000b0c5cf (table test-table [id=6663fb4d9e144fe88639cd0d24c729e7]) successfully deleted
I20260927 10:55:33.900903 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 11766
I20260927 10:55:33.907543 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 11404
2026-09-27T10:55:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2317 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11941](info): set up 2 sockets
Sep 27 10:55:33 dist-test-slave-cffh krb5kdc[11941](info): commencing operation
krb5kdc: starting...
W20260927 10:55:34.951931 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:55:35 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:55:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/rangeradmin_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/rangeradmin_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/rangerlookup_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/rangerlookup_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
I20260927 10:55:35.110391 1211 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZA5khf/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/postgres -l logfile start
2026-09-27 10:55:37.497 UTC [11971] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:55:37.497 UTC [11971] LOG: listening on IPv4 address "127.1.46.212", port 46375
2026-09-27 10:55:37.501 UTC [11971] LOG: listening on Unix socket "/tmp/.s.PGSQL.46375"
2026-09-27 10:55:37.534 UTC [11974] LOG: database system was shut down at 2026-09-27 10:55:36 UTC
2026-09-27 10:55:37.538 UTC [11971] LOG: database system is ready to accept connections
I20260927 10:55:38.471392 1211 mini_postgres.cc:96] Postgres bound to 46375
2026-09-27 10:55:38.477 UTC [11982] FATAL: database "slave" does not exist
127.1.46.212:46375 - accepting connections
I20260927 10:55:38.478225 1211 mini_ranger.cc:161] Starting Ranger...
I20260927 10:55:38.493063 1211 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:55:38.953712 1211 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:55:38.953809 1211 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-admin
2026-09-27 10:55:39,225 [I] DB FLAVOR :POSTGRES
2026-09-27 10:55:39,226 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:55:39,226 [I] Checking connection..
2026-09-27 10:55:39,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:55:39,475 [I] Checking connection passed.
2026-09-27 10:55:39,475 [I] --------- Verifying version history table ---------
2026-09-27 10:55:39,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:55:39,731 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:55:39,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:55:39,967 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:55:39,967 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:55:39,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:55:40.183 UTC [12046] WARNING: there is no transaction in progress
2026-09-27 10:55:40,189 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:55:40,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:55:40,439 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:55:40,440 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:55:40,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:55:40,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:55:40,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cffh.c.gcp-upstream.internal','N') ;"
2026-09-27 10:55:41,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:55:41,361 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:55:41,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:55:41,588 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:55:41,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:55:41,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:55:42,075 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:55:42,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:55:42.998 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.013 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.029 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.044 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.059 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.079 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.132 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.138 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.152 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.161 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.171 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.184 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.193 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.199 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.206 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43.212 UTC [12154] WARNING: there is no transaction in progress
2026-09-27 10:55:43,819 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:55:43,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:55:44,060 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:55:44,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:55:44,314 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:55:44,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:55:44,581 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:55:44,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:55:44,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:55:45,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cffh.c.gcp-upstream.internal';"
2026-09-27 10:55:45,303 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:55:45,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:55:45,558 [I] DB_PATCHES have already been applied
I20260927 10:55:45.564196 1211 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZA5khf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:55:45.564257 1211 mini_ranger.cc:193] Using host: 127.1.46.212
I20260927 10:55:45.567759 1211 mini_ranger.cc:227] Ranger admin URL: http://127.1.46.212:54541
Sep 27, 2026 10:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:55:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:55:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:55:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:55:46.656690 1211 mini_ranger.cc:160] Time spent starting Ranger: real 8.178s user 0.005s sys 0.005s
Sep 27, 2026 10:56:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:56:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:56:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:56:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:56:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:56:08.398085 1211 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/rangerkms_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/rangerkms_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:56:08 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506568, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:56:08.474889 1211 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:56:08.484180 1211 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:56:09.291203 1211 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:56:09.291280 1211 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms
I20260927 10:56:09.291368 1211 mini_ranger_kms.cc:227] Using postgres at 127.1.46.212:46375
2026-09-27 10:56:09,465 [I] DB FLAVOR :POSTGRES
2026-09-27 10:56:09,466 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:56:09,466 [I] Checking connection
2026-09-27 10:56:09,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:56:09,716 [I] connection success
2026-09-27 10:56:09,716 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:56:09,716 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:56:09,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:56:09,960 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:56:09,960 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:56:09,960 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:56:09,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:46375/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:56:10,202 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:56:10.617762 1211 mini_ranger_kms.cc:310] Created kms service
I20260927 10:56:10.805235 1211 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:56:10.845837 1211 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:56:10 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506568, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:56:11.227212 1211 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:56:11.227356 1211 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:56:11.227380 1211 mini_ranger_kms.cc:240] Using host: 127.1.46.212
I20260927 10:56:11.231850 1211 mini_ranger_kms.cc:276] Ranger KMS PID: 12334
I20260927 10:56:11.231987 1211 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.46.212:51113
Sep 27, 2026 10:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 10:56:13.449023 1211 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.974s user 0.015s sys 0.145s
I20260927 10:56:13.449163 1211 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:56:13.449318 1211 mini_ranger_kms.cc:401] 127.1.46.212:51113/kms/v1/keys
Sep 27 10:56:13 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506573, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 10:56:13 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506573, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.46.212@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
Sep 27 10:56:14 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506573, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.46.212@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
Sep 27 10:56:16 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:56:16.388775 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45253
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45253
--ranger_config_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:51113/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:16.495326 12385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:16.495580 12385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:16.495663 12385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:16.499611 12385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:56:16.499687 12385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:16.499739 12385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:16.499795 12385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:56:16.499835 12385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:56:16.505113 12385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.46.212:51113/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45253
--ranger_config_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:56:16.506045 12385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:16.507058 12385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:56:16.516584 12385 server_base.cc:1034] running on GCE node
W20260927 10:56:16.516551 12391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:16.516593 12393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:16.516577 12390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:16.517436 12385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:16.518392 12385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:16.519572 12385 hybrid_clock.cc:648] HybridClock initialized: now 1790506576519553 us; error 59 us; skew 500 ppm
Sep 27 10:56:16 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506576, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:16.527822 12385 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:56:16.528823 12385 webserver.cc:460] Webserver started at http://127.1.46.254:32819/ using document root <none> and password file <none>
I20260927 10:56:16.529331 12385 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:16.529407 12385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:16.529599 12385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:56:16 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506576, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:56:16.578013 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "ca6a1e186f0c4f799ec02c9d970df0ab"
format_stamp: "Formatted at 2026-09-27 10:56:16 on dist-test-slave-cffh"
server_key: "933b8a19775c25f58ab3de6d1c461ec3"
server_key_iv: "477767e4c778eedbd144cccab9d4fa33"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:16.578738 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "ca6a1e186f0c4f799ec02c9d970df0ab"
format_stamp: "Formatted at 2026-09-27 10:56:16 on dist-test-slave-cffh"
server_key: "933b8a19775c25f58ab3de6d1c461ec3"
server_key_iv: "477767e4c778eedbd144cccab9d4fa33"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:16.582881 12385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:56:16.600342 12400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:16.601804 12385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:56:16.601907 12385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "ca6a1e186f0c4f799ec02c9d970df0ab"
format_stamp: "Formatted at 2026-09-27 10:56:16 on dist-test-slave-cffh"
server_key: "933b8a19775c25f58ab3de6d1c461ec3"
server_key_iv: "477767e4c778eedbd144cccab9d4fa33"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:16.602088 12385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:16.619657 12385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:16.622336 12385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:16.622521 12385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:16.631023 12385 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45253
I20260927 10:56:16.631067 12452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45253 every 8 connection(s)
I20260927 10:56:16.631955 12385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:56:16.635418 12453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:16.636366 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 12385
I20260927 10:56:16.636555 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:56:16.636869 1211 external_mini_cluster.cc:1428] Setting key b911a0335d760fdfa099f447366c34e9
I20260927 10:56:16.642086 12453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab: Bootstrap starting.
I20260927 10:56:16.644574 12453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:16.645427 12453 log.cc:824] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:16.647372 12453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab: No bootstrap required, opened a new log
Sep 27 10:56:16 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:56:16.651876 12453 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45253 } }
I20260927 10:56:16.652134 12453 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:16.652225 12453 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6a1e186f0c4f799ec02c9d970df0ab, State: Initialized, Role: FOLLOWER
I20260927 10:56:16.652736 12453 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45253 } }
I20260927 10:56:16.652946 12453 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:56:16.653071 12453 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:56:16.653218 12453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:16.654230 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:16.638534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.1.46.254:45253)
0927 10:56:16.639337 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:16.639354 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:16.639392 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:56:16.640266 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:16.640288 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:16.640675 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:16.641923 (+ 1248us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:16.642316 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:16.643712 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:16.644945 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:16.645619 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:16.645852 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:16.648964 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:16.648971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:16.648992 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:16.649024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:16.651055 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:16.651550 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:16.651555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:16.651560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:16.651687 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:16.651992 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:16.651996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:16.651997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:16.652487 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:16.652694 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:16.652965 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:56:16.653229 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":100,"threads_started":1}
I20260927 10:56:16.654309 12453 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45253 } }
I20260927 10:56:16.654650 12453 leader_election.cc:302] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6a1e186f0c4f799ec02c9d970df0ab; no voters:
I20260927 10:56:16.654975 12453 leader_election.cc:288] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:56:16.655081 12458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:56:16.655408 12458 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [term 1 LEADER]: Becoming Leader. State: Replica: ca6a1e186f0c4f799ec02c9d970df0ab, State: Running, Role: LEADER
I20260927 10:56:16.655758 12458 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45253 } }
I20260927 10:56:16.656231 12453 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:56:16.657368 12458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6a1e186f0c4f799ec02c9d970df0ab. Latest consensus state: current_term: 1 leader_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45253 } } }
I20260927 10:56:16.657475 12458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:16.657600 12460 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6a1e186f0c4f799ec02c9d970df0ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45253 } } }
I20260927 10:56:16.657707 12460 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab [sys.catalog]: This master's current role is: LEADER
I20260927 10:56:16.658130 12467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:56:16.660894 12467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:56:16.661365 12453 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:56:16.674853 12467 catalog_manager.cc:1347] Generated new cluster ID: 57840f8ff9e14299aa9109d14d795a49
I20260927 10:56:16.674937 12467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:56:16.681025 12467 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:56:16.682410 12467 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:56:16.695596 12467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab: Generated new TSK 0
I20260927 10:56:16.696501 12467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:56:17.451838 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:51113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45253
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:17.591763 12498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:17.592064 12498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:17.592202 12498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:17.596446 12498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:17.596525 12498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:17.596630 12498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:56:17.602893 12498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.46.212:51113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:56:17.604475 12498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:17.605947 12498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:17.618692 12505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:17.619241 12504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:17.620253 12507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:17.623726 12498 server_base.cc:1034] running on GCE node
I20260927 10:56:17.624145 12498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:17.624845 12498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:17.626036 12498 hybrid_clock.cc:648] HybridClock initialized: now 1790506577625993 us; error 52 us; skew 500 ppm
Sep 27 10:56:17 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:17.629185 12498 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:56:17.630302 12498 webserver.cc:460] Webserver started at http://127.1.46.193:36643/ using document root <none> and password file <none>
I20260927 10:56:17.631106 12498 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:17.631270 12498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:17.631551 12498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:56:17 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:56:17.662066 12498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "5e9faaa7acf84d86b929d48344b56f99"
format_stamp: "Formatted at 2026-09-27 10:56:17 on dist-test-slave-cffh"
server_key: "17ddff4e15f3de8ba0f7beccfd520632"
server_key_iv: "d3173894a3b2b58872c2009a0cd9bbb4"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:17.663084 12498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "5e9faaa7acf84d86b929d48344b56f99"
format_stamp: "Formatted at 2026-09-27 10:56:17 on dist-test-slave-cffh"
server_key: "17ddff4e15f3de8ba0f7beccfd520632"
server_key_iv: "d3173894a3b2b58872c2009a0cd9bbb4"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:17.668640 12498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:56:17.685539 12515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:17.687171 12498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:56:17.687361 12498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "5e9faaa7acf84d86b929d48344b56f99"
format_stamp: "Formatted at 2026-09-27 10:56:17 on dist-test-slave-cffh"
server_key: "17ddff4e15f3de8ba0f7beccfd520632"
server_key_iv: "d3173894a3b2b58872c2009a0cd9bbb4"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:17.687580 12498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:17.707026 12498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:17.710871 12498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:17.711136 12498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:17.711989 12498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:17.713366 12498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:17.713479 12498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:17.713624 12498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:17.713719 12498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:17.729275 12498 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:39857
I20260927 10:56:17.730443 12498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:56:17.732430 12630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:39857 every 8 connection(s)
I20260927 10:56:17.734069 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 12498
I20260927 10:56:17.734205 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:56:17.734586 1211 external_mini_cluster.cc:1428] Setting key 3df7d5643fd9f4a18add94e6d7782c18
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Sep 27 10:56:17 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:56:17.785233 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:17.736109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:57429 (local address 127.1.46.254:45253)
0927 10:56:17.754526 (+ 18417us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:17.754531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:17.754549 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:56:17.754623 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:17.754627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:17.754693 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:17.754792 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:17.763094 (+ 8302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:17.764071 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:17.772443 (+ 8372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:17.772868 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:17.772976 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:17.780395 (+ 7419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:17.780400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:17.780403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:17.780436 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:17.782487 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:17.783233 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:17.783238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:17.783240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:17.783335 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:17.783797 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:17.783801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:17.783803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:17.784049 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:17.784142 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:17.784881 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 10:56:17.785021 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18331,"thread_start_us":60,"threads_started":1}
I20260927 10:56:17.787375 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:17.735004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45253 (local address 127.1.46.193:57429)
0927 10:56:17.752854 (+ 17850us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:17.752929 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:17.754046 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:17.761725 (+ 7679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:17.761737 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:17.762105 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:17.762876 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:17.762900 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:17.770678 (+ 7778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:17.770683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:17.772260 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:17.772273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:17.773235 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:17.773240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:17.773460 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:17.774348 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:17.774372 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:17.780148 (+ 5776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:17.782705 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:17.782713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:17.782733 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:17.783099 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:17.783454 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:17.783457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:17.783459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:17.783673 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:17.784216 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:17.784223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:17.784560 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 10:56:17.785894 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0927 10:56:17.786265 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17544,"thread_start_us":140,"threads_started":1}
I20260927 10:56:17.791277 12631 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45253
I20260927 10:56:17.791529 12631 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:17.792162 12631 heartbeater.cc:511] Master 127.1.46.254:45253 requested a full tablet report, sending...
I20260927 10:56:17.794094 12417 ts_manager.cc:194] Registered new tserver with Master: 5e9faaa7acf84d86b929d48344b56f99 (127.1.46.193:39857)
I20260927 10:56:17.795909 12417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:57429
Sep 27 10:56:17 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:56:17.817416 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:51113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45253
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:56:17.981920 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:17.982259 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:17.982322 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:17.988339 12637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:17.988418 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:17.988502 12637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:56:17.995200 12637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.46.212:51113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:56:17.996241 12637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:17.997534 12637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:18.008838 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:18.009080 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:18.008872 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:18.011694 12637 server_base.cc:1034] running on GCE node
I20260927 10:56:18.012162 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:18.012934 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:18.014117 12637 hybrid_clock.cc:648] HybridClock initialized: now 1790506578014096 us; error 37 us; skew 500 ppm
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:18.017717 12637 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:56:18.018888 12637 webserver.cc:460] Webserver started at http://127.1.46.194:35757/ using document root <none> and password file <none>
I20260927 10:56:18.019596 12637 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:18.019739 12637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:18.019964 12637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:56:18.041518 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "a5eeec31d13d4d2b9ed6499b9271345d"
format_stamp: "Formatted at 2026-09-27 10:56:18 on dist-test-slave-cffh"
server_key: "3bc153972698c09ea3306e53f42be335"
server_key_iv: "8097ce5f405c584ce58afd006c30643d"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:18.042239 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "a5eeec31d13d4d2b9ed6499b9271345d"
format_stamp: "Formatted at 2026-09-27 10:56:18 on dist-test-slave-cffh"
server_key: "3bc153972698c09ea3306e53f42be335"
server_key_iv: "8097ce5f405c584ce58afd006c30643d"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:18.047161 12637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 10:56:18.069875 12653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:18.079875 12637 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260927 10:56:18.080107 12637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "a5eeec31d13d4d2b9ed6499b9271345d"
format_stamp: "Formatted at 2026-09-27 10:56:18 on dist-test-slave-cffh"
server_key: "3bc153972698c09ea3306e53f42be335"
server_key_iv: "8097ce5f405c584ce58afd006c30643d"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:18.080372 12637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:18.109148 12637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:18.112978 12637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:18.113222 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:18.113977 12637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:18.115046 12637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:18.115146 12637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:18.115265 12637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:18.115329 12637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:18.129766 12637 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:44441
I20260927 10:56:18.131038 12637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:56:18.131601 12766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:44441 every 8 connection(s)
I20260927 10:56:18.140316 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 12637
I20260927 10:56:18.140443 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:56:18.140775 1211 external_mini_cluster.cc:1428] Setting key 11eb79bd0cb2eab4891a4479de01c91f
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:56:18.153141 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.135349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:43343 (local address 127.1.46.254:45253)
0927 10:56:18.135475 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.135480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.135998 (+ 518us) server_negotiation.cc:407] Connection header received
0927 10:56:18.137191 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.137196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.137255 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.137313 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:18.139167 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.139997 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.144979 (+ 4982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.145333 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.145411 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.148689 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:18.148705 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:18.148708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:18.148733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:18.150517 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:18.151160 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:18.151164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:18.151165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:18.151247 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:18.151619 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:18.151623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:18.151624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:18.151840 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:18.152110 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.152818 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.152952 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:56:18.155156 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.134269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45253 (local address 127.1.46.194:43343)
0927 10:56:18.135831 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.135902 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:18.136931 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:18.137726 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:18.137734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:18.138083 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:18.138789 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.138808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.141331 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.141334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.144839 (+ 3505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.144850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.145610 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.145614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.145861 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.146735 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:18.146757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:18.148499 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:18.150690 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:18.150707 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:18.150723 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:18.151046 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:18.151348 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:18.151351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:18.151353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:18.151516 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:18.152217 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:18.152224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:18.152525 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 10:56:18.153604 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0927 10:56:18.153962 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1358,"thread_start_us":143,"threads_started":1}
I20260927 10:56:18.159386 12767 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45253
I20260927 10:56:18.159636 12767 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:18.160221 12767 heartbeater.cc:511] Master 127.1.46.254:45253 requested a full tablet report, sending...
I20260927 10:56:18.161393 12417 ts_manager.cc:194] Registered new tserver with Master: a5eeec31d13d4d2b9ed6499b9271345d (127.1.46.194:44441)
I20260927 10:56:18.162139 12417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:43343
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:56:18.195565 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:51113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45253
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
W20260927 10:56:18.357208 12772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:56:18.357672 12772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:56:18.357815 12772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:56:18.365269 12772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:56:18.365402 12772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:56:18.365532 12772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:56:18.372969 12772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.46.212:51113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:56:18.374436 12772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:56:18.375851 12772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:56:18.388360 12777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:56:18.388717 12780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:18.391736 12772 server_base.cc:1034] running on GCE node
W20260927 10:56:18.394784 12778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:56:18.395283 12772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:56:18.396019 12772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:56:18.397187 12772 hybrid_clock.cc:648] HybridClock initialized: now 1790506578397127 us; error 75 us; skew 500 ppm
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:18.400552 12772 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:56:18.401496 12772 webserver.cc:460] Webserver started at http://127.1.46.195:33775/ using document root <none> and password file <none>
I20260927 10:56:18.402138 12772 fs_manager.cc:359] Metadata directory not provided
I20260927 10:56:18.402202 12772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:56:18.402397 12772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:56:18.422943 12772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "cb08399e6ce54230b8960be2665dcf17"
format_stamp: "Formatted at 2026-09-27 10:56:18 on dist-test-slave-cffh"
server_key: "e1c7cef0d79b83be60db95c26cdae0f5"
server_key_iv: "f67bf6341d91e099b63f07b16526636d"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:18.424674 12772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "cb08399e6ce54230b8960be2665dcf17"
format_stamp: "Formatted at 2026-09-27 10:56:18 on dist-test-slave-cffh"
server_key: "e1c7cef0d79b83be60db95c26cdae0f5"
server_key_iv: "f67bf6341d91e099b63f07b16526636d"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:18.429554 12772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 10:56:18.455238 12787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:18.456868 12772 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:56:18.457026 12772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "cb08399e6ce54230b8960be2665dcf17"
format_stamp: "Formatted at 2026-09-27 10:56:18 on dist-test-slave-cffh"
server_key: "e1c7cef0d79b83be60db95c26cdae0f5"
server_key_iv: "f67bf6341d91e099b63f07b16526636d"
server_key_version: "kuduclusterkey@0"
I20260927 10:56:18.457186 12772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:56:18.472954 12772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:56:18.475945 12772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:56:18.476140 12772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:56:18.476760 12772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:56:18.477617 12772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:56:18.477685 12772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:18.477794 12772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:56:18.477846 12772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:56:18.489957 12772 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:38947
I20260927 10:56:18.490192 12900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:38947 every 8 connection(s)
I20260927 10:56:18.491240 12772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:56:18.498183 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 12772
I20260927 10:56:18.498322 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:56:18.498756 1211 external_mini_cluster.cc:1428] Setting key cbede4dafdb1a9944af1bfe846f0cadf
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:56:18.508740 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.493588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:36521 (local address 127.1.46.254:45253)
0927 10:56:18.493767 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.493773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.495451 (+ 1678us) server_negotiation.cc:407] Connection header received
0927 10:56:18.496612 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.496616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.496692 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.496820 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:18.498229 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.499130 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.500941 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.501258 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.501385 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.504415 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:18.504419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:18.504422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:18.504449 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:18.506261 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:18.506949 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:18.506954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:18.506955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:18.507041 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:18.507462 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:18.507465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:18.507467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:18.507678 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:18.507777 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.508438 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.508561 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:56:18.510552 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.494105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45253 (local address 127.1.46.195:36521)
0927 10:56:18.495265 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.495318 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:18.496353 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:18.497014 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:18.497024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:18.497370 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:56:18.498050 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.498070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.499380 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.499383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.500804 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.500817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.501585 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.501588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.501800 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.502478 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:56:18.502498 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:56:18.504226 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:56:18.506400 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:18.506405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:18.506420 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:18.506822 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:18.507164 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:56:18.507167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:56:18.507169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:56:18.507342 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:56:18.507846 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:56:18.507852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:56:18.508150 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 10:56:18.509179 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0927 10:56:18.509508 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":118,"threads_started":1}
I20260927 10:56:18.512180 12901 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45253
I20260927 10:56:18.512424 12901 heartbeater.cc:464] Registering TS with master...
I20260927 10:56:18.512975 12901 heartbeater.cc:511] Master 127.1.46.254:45253 requested a full tablet report, sending...
I20260927 10:56:18.514654 12416 ts_manager.cc:194] Registered new tserver with Master: cb08399e6ce54230b8960be2665dcf17 (127.1.46.195:38947)
I20260927 10:56:18.515439 12416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:36521
I20260927 10:56:18.516199 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:56:18.529175 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.518526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.1.46.254:45253)
0927 10:56:18.518675 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.518679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.519712 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 10:56:18.519894 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.519898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.519943 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.519998 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:56:18.520972 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.521762 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.523006 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.523341 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.523419 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.525224 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:56:18.525227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:56:18.525232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:56:18.525258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:56:18.527060 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:18.527772 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:18.527776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:18.527778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:18.527858 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:56:18.528182 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:56:18.528186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:56:18.528188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:56:18.528413 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:56:18.528526 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.528918 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.529029 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:56:18.532963 12417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:56:18.537407 12417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:56:18.572523 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.551091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:38947 (local address 127.0.0.1:53044)
0927 10:56:18.554466 (+ 3375us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.554479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:18.554557 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:18.557290 (+ 2733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:18.557297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:18.557322 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:18.557444 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.557452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.567042 (+ 9590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.567046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.568913 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.568924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.571942 (+ 3018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.571958 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.572072 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.572110 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:56:18.572242 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 10:56:18.572366 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3310,"thread_start_us":82,"threads_started":1}
I20260927 10:56:18.573944 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.552255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:44441 (local address 127.0.0.1:34126)
0927 10:56:18.555574 (+ 3319us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.555587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:18.556596 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:18.558294 (+ 1698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:18.558297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:18.558316 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:18.558401 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.558407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.563310 (+ 4903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.563314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.564762 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.564772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.573537 (+ 8765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.573541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.573630 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.573651 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:56:18.573738 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:56:18.573806 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3261,"thread_start_us":80,"threads_started":1}
I20260927 10:56:18.576741 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.548693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39857 (local address 127.0.0.1:36614)
0927 10:56:18.554077 (+ 5384us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.554125 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:18.554246 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:18.558519 (+ 4273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:18.558522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:18.558539 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:18.558659 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.558665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.565500 (+ 6835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.565503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.569342 (+ 3839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.569352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.576327 (+ 6975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.576331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.576424 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.576446 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:56:18.576503 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:56:18.576570 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5187,"thread_start_us":94,"threads_started":1}
I20260927 10:56:18.577415 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.551497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.1.46.195:38947)
0927 10:56:18.554836 (+ 3339us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.554850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.554874 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:56:18.554948 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.554954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.555167 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.555329 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:18.559541 (+ 4212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.566898 (+ 7357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.570803 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.571602 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.571727 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.571872 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.577162 (+ 5290us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.577276 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3232,"thread_start_us":143,"threads_started":1}
I20260927 10:56:18.577944 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.552627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.1.46.194:44441)
0927 10:56:18.556516 (+ 3889us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.556527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.556551 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:56:18.556786 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.556793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.556980 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.557124 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:18.558799 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.562486 (+ 3687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.569580 (+ 7094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.570421 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.570528 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.570720 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.577661 (+ 6941us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.577788 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3786,"thread_start_us":172,"threads_started":1}
I20260927 10:56:18.581380 12817 tablet_service.cc:1467] Processing CreateTablet for tablet c0501552ad6a48bfb8605d69b600e039 (DEFAULT_TABLE table=test-table [id=29ae7325d35845329faa2f55b8b4cef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:18.582543 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0501552ad6a48bfb8605d69b600e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:18.593742 12701 tablet_service.cc:1467] Processing CreateTablet for tablet c0501552ad6a48bfb8605d69b600e039 (DEFAULT_TABLE table=test-table [id=29ae7325d35845329faa2f55b8b4cef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:18.596160 12701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0501552ad6a48bfb8605d69b600e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:18.607012 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.549799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.1.46.193:39857)
0927 10:56:18.557740 (+ 7941us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.557746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.557769 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:56:18.557855 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.557860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.558019 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.558147 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:18.560917 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.565364 (+ 4447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.574027 (+ 8663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.575946 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.576048 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.606646 (+ 30598us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.606715 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.606838 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7806,"thread_start_us":103,"threads_started":1}
I20260927 10:56:18.616422 12563 tablet_service.cc:1467] Processing CreateTablet for tablet c0501552ad6a48bfb8605d69b600e039 (DEFAULT_TABLE table=test-table [id=29ae7325d35845329faa2f55b8b4cef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:56:18.617826 12563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0501552ad6a48bfb8605d69b600e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:56:18.638350 12920 tablet_bootstrap.cc:492] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Bootstrap starting.
I20260927 10:56:18.639153 12921 tablet_bootstrap.cc:492] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Bootstrap starting.
I20260927 10:56:18.641055 12920 tablet_bootstrap.cc:654] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:18.642422 12920 log.cc:824] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:18.646260 12921 tablet_bootstrap.cc:654] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:18.647192 12921 log.cc:824] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:18.647686 12922 tablet_bootstrap.cc:492] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99: Bootstrap starting.
I20260927 10:56:18.654754 12922 tablet_bootstrap.cc:654] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99: Neither blocks nor log segments found. Creating new log.
I20260927 10:56:18.656196 12922 log.cc:824] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99: Log is configured to *not* fsync() on all Append() calls
I20260927 10:56:18.657939 12921 tablet_bootstrap.cc:492] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: No bootstrap required, opened a new log
I20260927 10:56:18.660249 12922 tablet_bootstrap.cc:492] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99: No bootstrap required, opened a new log
I20260927 10:56:18.660362 12921 ts_tablet_manager.cc:1397] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20260927 10:56:18.665627 12922 ts_tablet_manager.cc:1397] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20260927 10:56:18.666709 12920 tablet_bootstrap.cc:492] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: No bootstrap required, opened a new log
I20260927 10:56:18.666920 12920 ts_tablet_manager.cc:1397] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20260927 10:56:18.669888 12921 raft_consensus.cc:348] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.670186 12921 raft_consensus.cc:374] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:18.670265 12921 raft_consensus.cc:729] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb08399e6ce54230b8960be2665dcf17, State: Initialized, Role: FOLLOWER
I20260927 10:56:18.670743 12920 raft_consensus.cc:348] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.671002 12920 raft_consensus.cc:374] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:18.671056 12920 raft_consensus.cc:729] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5eeec31d13d4d2b9ed6499b9271345d, State: Initialized, Role: FOLLOWER
I20260927 10:56:18.671710 12920 consensus_queue.cc:260] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.671794 12922 raft_consensus.cc:348] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.672405 12921 consensus_queue.cc:260] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.672827 12920 ts_tablet_manager.cc:1428] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:56:18.673260 12767 heartbeater.cc:503] Master 127.1.46.254:45253 was elected leader, sending a full tablet report...
I20260927 10:56:18.678238 12921 ts_tablet_manager.cc:1428] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260927 10:56:18.678670 12901 heartbeater.cc:503] Master 127.1.46.254:45253 was elected leader, sending a full tablet report...
I20260927 10:56:18.678639 12922 raft_consensus.cc:374] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:56:18.678785 12922 raft_consensus.cc:729] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9faaa7acf84d86b929d48344b56f99, State: Initialized, Role: FOLLOWER
I20260927 10:56:18.679286 12922 consensus_queue.cc:260] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.680343 12922 ts_tablet_manager.cc:1428] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260927 10:56:18.680787 12631 heartbeater.cc:503] Master 127.1.46.254:45253 was elected leader, sending a full tablet report...
I20260927 10:56:18.709141 12926 raft_consensus.cc:482] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:56:18.709362 12926 raft_consensus.cc:504] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.710752 12926 leader_election.cc:288] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9faaa7acf84d86b929d48344b56f99 (127.1.46.193:39857), cb08399e6ce54230b8960be2665dcf17 (127.1.46.195:38947)
Sep 27 10:56:18 dist-test-slave-cffh krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:56:18.721875 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.711989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:57479 (local address 127.1.46.193:39857)
0927 10:56:18.712222 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.712227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.712242 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:56:18.712295 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.712299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.712343 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.712435 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:18.714005 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.718797 (+ 4792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.720680 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.721281 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.721441 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.721487 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.721664 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.721753 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260927 10:56:18.721894 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.711156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39857 (local address 127.1.46.194:57479)
0927 10:56:18.711466 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.711481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:18.711585 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:18.712444 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:18.712447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:18.712461 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:18.712588 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.712593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.718904 (+ 6311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.718907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.720188 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.720199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.721416 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.721437 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.721546 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.721572 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:56:18.721645 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:56:18.721718 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":124,"threads_started":1}
I20260927 10:56:18.723114 12578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0501552ad6a48bfb8605d69b600e039" candidate_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9faaa7acf84d86b929d48344b56f99" is_pre_election: true
I20260927 10:56:18.723454 12578 raft_consensus.cc:2454] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5eeec31d13d4d2b9ed6499b9271345d in term 0.
I20260927 10:56:18.724121 12654 leader_election.cc:302] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9faaa7acf84d86b929d48344b56f99, a5eeec31d13d4d2b9ed6499b9271345d; no voters:
I20260927 10:56:18.724406 12926 raft_consensus.cc:2781] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:56:18.724496 12926 raft_consensus.cc:482] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:56:18.724546 12926 raft_consensus.cc:3037] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:18.725540 12926 raft_consensus.cc:504] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.725878 12926 leader_election.cc:288] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [CANDIDATE]: Term 1 election: Requested vote from peers 5e9faaa7acf84d86b929d48344b56f99 (127.1.46.193:39857), cb08399e6ce54230b8960be2665dcf17 (127.1.46.195:38947)
I20260927 10:56:18.726147 12578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0501552ad6a48bfb8605d69b600e039" candidate_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9faaa7acf84d86b929d48344b56f99"
I20260927 10:56:18.726260 12578 raft_consensus.cc:3037] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:56:18.727538 12578 raft_consensus.cc:2454] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5eeec31d13d4d2b9ed6499b9271345d in term 1.
I20260927 10:56:18.727945 12654 leader_election.cc:302] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9faaa7acf84d86b929d48344b56f99, a5eeec31d13d4d2b9ed6499b9271345d; no voters:
I20260927 10:56:18.728178 12926 raft_consensus.cc:2781] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:56:18.728430 12926 raft_consensus.cc:686] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 1 LEADER]: Becoming Leader. State: Replica: a5eeec31d13d4d2b9ed6499b9271345d, State: Running, Role: LEADER
I20260927 10:56:18.728820 12926 consensus_queue.cc:237] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } }
I20260927 10:56:18.732398 12417 catalog_manager.cc:5495] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d reported cstate change: term changed from 0 to 1, leader changed from <none> to a5eeec31d13d4d2b9ed6499b9271345d (127.1.46.194). New cstate: current_term: 1 leader_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 44441 } health_report { overall_health: HEALTHY } } }
W20260927 10:56:18.738344 12632 tablet.cc:2369] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:56:18.739585 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.712103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:39213 (local address 127.1.46.195:38947)
0927 10:56:18.712608 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.712613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.712626 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:56:18.712674 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.712676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.712722 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.712801 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:56:18.713030 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.713833 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.718944 (+ 5111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.719512 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.719635 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.719689 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.739358 (+ 19669us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.739463 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423}
I20260927 10:56:18.740125 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.711561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:38947 (local address 127.1.46.194:39213)
0927 10:56:18.711838 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.711850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:56:18.711922 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:56:18.712806 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:56:18.712808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:56:18.712823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:56:18.712932 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.712938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.713957 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.713961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.715717 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:56:18.715730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.739037 (+ 23307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.739042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:56:18.739166 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.739197 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:56:18.739907 (+ 710us) client_negotiation.cc:240] Negotiation successful
0927 10:56:18.739994 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260927 10:56:18.740967 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0501552ad6a48bfb8605d69b600e039" candidate_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb08399e6ce54230b8960be2665dcf17" is_pre_election: true
I20260927 10:56:18.741230 12855 raft_consensus.cc:2454] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5eeec31d13d4d2b9ed6499b9271345d in term 0.
I20260927 10:56:18.741528 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0501552ad6a48bfb8605d69b600e039" candidate_uuid: "a5eeec31d13d4d2b9ed6499b9271345d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb08399e6ce54230b8960be2665dcf17"
I20260927 10:56:18.741616 12844 raft_consensus.cc:3037] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:56:18.742338 12902 tablet.cc:2369] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:56:18.742861 12844 raft_consensus.cc:2454] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5eeec31d13d4d2b9ed6499b9271345d in term 1.
I20260927 10:56:18.768939 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.763615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.1.46.194:44441)
0927 10:56:18.763735 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:56:18.763740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:56:18.763755 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:56:18.763915 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:56:18.763919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:56:18.763993 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:56:18.764105 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:56:18.764526 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.765346 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.766704 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:56:18.767064 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:56:18.767170 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:56:18.767617 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:56:18.768183 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:56:18.768304 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:56:18.768723 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 10:56:18.768783 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:56:18.775698 12844 raft_consensus.cc:1261] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 1 FOLLOWER]: Refusing update from remote peer a5eeec31d13d4d2b9ed6499b9271345d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:56:18.775839 12578 raft_consensus.cc:1261] T c0501552ad6a48bfb8605d69b600e039 P 5e9faaa7acf84d86b929d48344b56f99 [term 1 FOLLOWER]: Refusing update from remote peer a5eeec31d13d4d2b9ed6499b9271345d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:56:18.776592 12932 consensus_queue.cc:1035] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9faaa7acf84d86b929d48344b56f99" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:56:18.776752 12926 consensus_queue.cc:1035] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb08399e6ce54230b8960be2665dcf17" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:56:18.788316 12936 mvcc.cc:205] Tried to move back new op lower bound from 7333914946657640448 to 7333914946476666880. Current Snapshot: MvccSnapshot[applied={T|T < 7333914946657640448}]
I20260927 10:56:18.790050 12938 mvcc.cc:205] Tried to move back new op lower bound from 7333914946657640448 to 7333914946476666880. Current Snapshot: MvccSnapshot[applied={T|T < 7333914946657640448}]
I20260927 10:56:18.790794 12937 mvcc.cc:205] Tried to move back new op lower bound from 7333914946657640448 to 7333914946476666880. Current Snapshot: MvccSnapshot[applied={T|T < 7333914946657640448}]
I20260927 10:56:18.800158 12417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:56:18.800333 12417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:56:18.803146 12417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca6a1e186f0c4f799ec02c9d970df0ab: Sending DeleteTablet for 3 replicas of tablet c0501552ad6a48bfb8605d69b600e039
I20260927 10:56:18.803992 12563 tablet_service.cc:1514] Processing DeleteTablet for tablet c0501552ad6a48bfb8605d69b600e039 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:18 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:36614
I20260927 10:56:18.804330 12817 tablet_service.cc:1514] Processing DeleteTablet for tablet c0501552ad6a48bfb8605d69b600e039 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:18 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:53044
I20260927 10:56:18.804811 12946 tablet_replica.cc:330] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: stopping tablet replica
I20260927 10:56:18.804796 12701 tablet_service.cc:1514] Processing DeleteTablet for tablet c0501552ad6a48bfb8605d69b600e039 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:56:18 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:34126
I20260927 10:56:18.805084 12946 raft_consensus.cc:2229] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:56:18.805401 12946 raft_consensus.cc:2258] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:56:18.806218 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 12498
I20260927 10:56:18.807021 12946 ts_tablet_manager.cc:1905] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:56:18.809747 12946 ts_tablet_manager.cc:1918] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:56:18.809880 12946 log.cc:1196] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/c0501552ad6a48bfb8605d69b600e039
I20260927 10:56:18.810554 12946 ts_tablet_manager.cc:1939] T c0501552ad6a48bfb8605d69b600e039 P cb08399e6ce54230b8960be2665dcf17: Deleting consensus metadata
I20260927 10:56:18.813504 12402 catalog_manager.cc:4841] TS cb08399e6ce54230b8960be2665dcf17 (127.1.46.195:38947): tablet c0501552ad6a48bfb8605d69b600e039 (table test-table [id=29ae7325d35845329faa2f55b8b4cef9]) successfully deleted
I20260927 10:56:18.825519 12947 tablet_replica.cc:330] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: stopping tablet replica
I20260927 10:56:18.825951 12947 raft_consensus.cc:2229] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:56:18.826485 12947 raft_consensus.cc:2258] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:56:18.832518 12947 ts_tablet_manager.cc:1905] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:56:18.835397 12947 ts_tablet_manager.cc:1918] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:56:18.835547 12947 log.cc:1196] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/c0501552ad6a48bfb8605d69b600e039
I20260927 10:56:18.835929 12947 ts_tablet_manager.cc:1939] T c0501552ad6a48bfb8605d69b600e039 P a5eeec31d13d4d2b9ed6499b9271345d: Deleting consensus metadata
I20260927 10:56:18.837073 12404 catalog_manager.cc:4841] TS a5eeec31d13d4d2b9ed6499b9271345d (127.1.46.194:44441): tablet c0501552ad6a48bfb8605d69b600e039 (table test-table [id=29ae7325d35845329faa2f55b8b4cef9]) successfully deleted
W20260927 10:56:18.837265 12401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:39857 (error 108)
I20260927 10:56:18.838392 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:56:18.838048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39857 (local address 127.0.0.1:36624)
0927 10:56:18.838235 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 10:56:18.838299 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 10:56:18.838552 12401 catalog_manager.cc:4584] TS 5e9faaa7acf84d86b929d48344b56f99 (127.1.46.193:39857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0501552ad6a48bfb8605d69b600e039: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39857: connect: Connection refused (error 111)
I20260927 10:56:18.838867 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 12637
I20260927 10:56:18.844208 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 12772
I20260927 10:56:18.851147 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 12385
I20260927 10:56:18.857549 1211 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:56:19.202270 1211 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:56:19.202392 1211 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:56:19.546694 1211 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:56:19.546 UTC [11971] LOG: received smart shutdown request
2026-09-27 10:56:19.550 UTC [11971] LOG: background worker "logical replication launcher" (PID 11980) exited with exit code 1
2026-09-27 10:56:19.550 UTC [11975] LOG: shutting down
2026-09-27 10:56:19.710 UTC [11971] LOG: database system is shut down
2026-09-27T10:56:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45921 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:56:19 dist-test-slave-cffh krb5kdc[12957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:56:19 dist-test-slave-cffh krb5kdc[12957](info): set up 2 sockets
Sep 27 10:56:19 dist-test-slave-cffh krb5kdc[12957](info): commencing operation
krb5kdc: starting...
W20260927 10:56:20.922750 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:56:20 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:56:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/rangeradmin_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/rangeradmin_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/rangerlookup_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/rangerlookup_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
I20260927 10:56:21.075042 1211 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZA5khf/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/postgres -l logfile start
2026-09-27 10:56:23.409 UTC [12987] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:56:23.409 UTC [12987] LOG: listening on IPv4 address "127.1.46.212", port 59187
2026-09-27 10:56:23.412 UTC [12987] LOG: listening on Unix socket "/tmp/.s.PGSQL.59187"
2026-09-27 10:56:23.432 UTC [12990] LOG: database system was shut down at 2026-09-27 10:56:22 UTC
2026-09-27 10:56:23.435 UTC [12987] LOG: database system is ready to accept connections
I20260927 10:56:24.386431 1211 mini_postgres.cc:96] Postgres bound to 59187
2026-09-27 10:56:24.393 UTC [12998] FATAL: database "slave" does not exist
127.1.46.212:59187 - accepting connections
I20260927 10:56:24.393540 1211 mini_ranger.cc:161] Starting Ranger...
I20260927 10:56:24.408581 1211 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:56:24.853096 1211 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:56:24.853183 1211 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-admin
2026-09-27 10:56:25,134 [I] DB FLAVOR :POSTGRES
2026-09-27 10:56:25,134 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:56:25,134 [I] Checking connection..
2026-09-27 10:56:25,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:56:25,374 [I] Checking connection passed.
2026-09-27 10:56:25,374 [I] --------- Verifying version history table ---------
2026-09-27 10:56:25,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:56:25,624 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:56:25,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:56:25,882 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:56:25,882 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:56:25,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:56:26.121 UTC [13062] WARNING: there is no transaction in progress
2026-09-27 10:56:26,127 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:56:26,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:56:26,365 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:56:26,366 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:56:26,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:56:26,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:56:26,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cffh.c.gcp-upstream.internal','N') ;"
2026-09-27 10:56:27,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:56:27,330 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:56:27,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:56:27,561 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:56:27,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:56:27,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:56:28,031 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:56:28,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:56:28.945 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:28.960 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:28.975 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:28.989 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.006 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.022 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.072 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.078 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.091 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.100 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.110 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.123 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.132 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.138 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.144 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29.151 UTC [13170] WARNING: there is no transaction in progress
2026-09-27 10:56:29,747 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:56:29,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:56:29,964 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:56:29,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:56:30,210 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:56:30,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:56:30,467 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:56:30,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:56:30,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:56:30,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cffh.c.gcp-upstream.internal';"
2026-09-27 10:56:31,207 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:56:31,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:56:31,473 [I] DB_PATCHES have already been applied
I20260927 10:56:31.478734 1211 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZA5khf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:56:31.478790 1211 mini_ranger.cc:193] Using host: 127.1.46.212
I20260927 10:56:31.482173 1211 mini_ranger.cc:227] Ranger admin URL: http://127.1.46.212:40743
Sep 27, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:56:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:56:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:56:32.609210 1211 mini_ranger.cc:160] Time spent starting Ranger: real 8.216s user 0.003s sys 0.007s
Sep 27, 2026 10:56:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:56:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:56:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:56:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:56:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:56:52.801349 1211 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/rangerkms_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/rangerkms_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/HTTP_127.1.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:56:52 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506612, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:56:52.881336 1211 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:56:52.890208 1211 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:56:53.684700 1211 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:56:53.684798 1211 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms
I20260927 10:56:53.684819 1211 mini_ranger_kms.cc:227] Using postgres at 127.1.46.212:59187
2026-09-27 10:56:53,853 [I] DB FLAVOR :POSTGRES
2026-09-27 10:56:53,854 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:56:53,854 [I] Checking connection
2026-09-27 10:56:53,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:56:54,076 [I] connection success
2026-09-27 10:56:54,077 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:56:54,077 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:56:54,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:56:54,306 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:56:54,306 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:56:54,306 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:56:54,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.46.212:59187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:56:54,575 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:56:55.030382 1211 mini_ranger_kms.cc:310] Created kms service
I20260927 10:56:55.218585 1211 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:56:55.261430 1211 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:56:55 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506612, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:56:55.649263 1211 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:56:55.649427 1211 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZA5khf/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZA5khf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:56:55.649452 1211 mini_ranger_kms.cc:240] Using host: 127.1.46.212
I20260927 10:56:55.654143 1211 mini_ranger_kms.cc:276] Ranger KMS PID: 13350
I20260927 10:56:55.654238 1211 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.46.212:39755
Sep 27, 2026 10:56:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:56:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:56:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:56:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 10:56:57 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:56:57.991147 1211 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.110s user 0.007s sys 0.146s
I20260927 10:56:57.991318 1211 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:56:57.991395 1211 mini_ranger_kms.cc:401] 127.1.46.212:39755/kms/v1/keys
Sep 27 10:56:58 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.46.212@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
Sep 27 10:56:58 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.46.212@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
Sep 27 10:57:00 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:57:00.808923 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45789
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45789
--ranger_config_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:00.921279 13401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:00.921545 13401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:00.921636 13401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:00.925554 13401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:00.925639 13401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:00.925683 13401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:00.925735 13401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:00.925786 13401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:00.931107 13401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.46.212:39755/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45789
--ranger_config_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:00.932138 13401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:00.933102 13401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:00.942272 13407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:00.942239 13406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:00.942240 13409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:00.942534 13401 server_base.cc:1034] running on GCE node
I20260927 10:57:00.943205 13401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:00.944419 13401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:00.945595 13401 hybrid_clock.cc:648] HybridClock initialized: now 1790506620945571 us; error 42 us; skew 500 ppm
Sep 27 10:57:00 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:00.948391 13401 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:00.949347 13401 webserver.cc:460] Webserver started at http://127.1.46.254:33675/ using document root <none> and password file <none>
I20260927 10:57:00.949919 13401 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:00.949972 13401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:00.950181 13401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:57:00 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:01.009022 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "075b114aa3a940948fd5c0fa02b5f578"
format_stamp: "Formatted at 2026-09-27 10:57:01 on dist-test-slave-cffh"
server_key: "df7c75532e1b9ee2ee8bcfad035d0db1"
server_key_iv: "09ae16381f3fed97193f6f43467dbda8"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:01.009651 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "075b114aa3a940948fd5c0fa02b5f578"
format_stamp: "Formatted at 2026-09-27 10:57:01 on dist-test-slave-cffh"
server_key: "df7c75532e1b9ee2ee8bcfad035d0db1"
server_key_iv: "09ae16381f3fed97193f6f43467dbda8"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:01.013465 13401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 10:57:01.026830 13416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:01.028227 13401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:57:01.028368 13401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "075b114aa3a940948fd5c0fa02b5f578"
format_stamp: "Formatted at 2026-09-27 10:57:01 on dist-test-slave-cffh"
server_key: "df7c75532e1b9ee2ee8bcfad035d0db1"
server_key_iv: "09ae16381f3fed97193f6f43467dbda8"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:01.028499 13401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:01.055658 13401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:01.058444 13401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:01.058672 13401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:01.066468 13401 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45789
I20260927 10:57:01.066507 13468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45789 every 8 connection(s)
I20260927 10:57:01.067407 13401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:01.070664 13469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:01.075762 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 13401
I20260927 10:57:01.075908 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:57:01.076191 1211 external_mini_cluster.cc:1428] Setting key f5565f790431b4c8c4a1e5872977279b
I20260927 10:57:01.077397 13469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Bootstrap starting.
I20260927 10:57:01.080111 13469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:01.080946 13469 log.cc:824] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:01.083040 13469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: No bootstrap required, opened a new log
Sep 27 10:57:01 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:01.087359 13469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:01.087594 13469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:01.087692 13469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075b114aa3a940948fd5c0fa02b5f578, State: Initialized, Role: FOLLOWER
I20260927 10:57:01.088186 13469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:01.088333 13469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:01.088411 13469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:01.088526 13469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:01.089541 13469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:01.089902 13469 leader_election.cc:302] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 075b114aa3a940948fd5c0fa02b5f578; no voters:
I20260927 10:57:01.090278 13469 leader_election.cc:288] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:01.090724 13474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:01.090945 13474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 LEADER]: Becoming Leader. State: Replica: 075b114aa3a940948fd5c0fa02b5f578, State: Running, Role: LEADER
I20260927 10:57:01.091279 13474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:01.091642 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:01.077612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.1.46.254:45789)
0927 10:57:01.077984 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:01.078004 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:01.078036 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:57:01.078683 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:01.078700 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:01.079043 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:01.079356 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:01.080152 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:01.081234 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:01.082279 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:01.083007 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:01.083181 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:01.086203 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:01.086210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:01.086226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:01.086256 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:01.088234 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:01.088775 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:01.088780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:01.088786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:01.088905 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:01.089270 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:01.089274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:01.089275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:01.089780 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:01.089968 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:01.090291 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 10:57:01.090637 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260927 10:57:01.091941 13469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:01.092430 13474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 075b114aa3a940948fd5c0fa02b5f578. Latest consensus state: current_term: 1 leader_uuid: "075b114aa3a940948fd5c0fa02b5f578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } } }
I20260927 10:57:01.092523 13474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:01.094178 13469 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:57:01.094715 13475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "075b114aa3a940948fd5c0fa02b5f578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } } }
I20260927 10:57:01.094827 13475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:01.095096 13478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:01.100965 13478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:01.111660 13478 catalog_manager.cc:1347] Generated new cluster ID: d23e20eb7aca49d1b5dcd181373b49a8
I20260927 10:57:01.111726 13478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:01.132490 13478 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:01.133726 13478 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:01.150753 13478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Generated new TSK 0
I20260927 10:57:01.151643 13478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:57:01.817795 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45789
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:01.975237 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:01.975576 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:01.975667 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:01.981741 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:01.981815 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:01.981916 13514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:57:01.989900 13514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.46.212:39755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:01.991235 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:01.992539 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:02.003520 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:02.003818 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:02.003311 13519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:02.004568 13514 server_base.cc:1034] running on GCE node
I20260927 10:57:02.005033 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:02.005888 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:02.007118 13514 hybrid_clock.cc:648] HybridClock initialized: now 1790506622007038 us; error 95 us; skew 500 ppm
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:02.010531 13514 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:57:02.011540 13514 webserver.cc:460] Webserver started at http://127.1.46.193:35117/ using document root <none> and password file <none>
I20260927 10:57:02.012243 13514 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:02.012331 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:02.012630 13514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:02.028695 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "afd325f3f02f4e3c987e463d3748b0ee"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "49c9163a9a56febfb9e9bb5bdab9acf8"
server_key_iv: "b589faf0b6d1ed55d0341458965fe1cf"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.029767 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "afd325f3f02f4e3c987e463d3748b0ee"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "49c9163a9a56febfb9e9bb5bdab9acf8"
server_key_iv: "b589faf0b6d1ed55d0341458965fe1cf"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.034687 13514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:57:02.053754 13531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.055269 13514 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260927 10:57:02.055423 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "afd325f3f02f4e3c987e463d3748b0ee"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "49c9163a9a56febfb9e9bb5bdab9acf8"
server_key_iv: "b589faf0b6d1ed55d0341458965fe1cf"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.055574 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:02.074851 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:02.078372 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:02.078653 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:02.079372 13514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:02.080381 13514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:02.080489 13514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.080626 13514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:02.080717 13514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.091298 13514 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:38305
I20260927 10:57:02.092509 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:02.102669 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 13514
I20260927 10:57:02.102787 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:02.103093 1211 external_mini_cluster.cc:1428] Setting key 63e33c10b07cd49593c39171f09386d2
I20260927 10:57:02.111722 13644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:38305 every 8 connection(s)
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:57:02.131978 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.114081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:46217 (local address 127.1.46.254:45789)
0927 10:57:02.115801 (+ 1720us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:02.115823 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:02.116442 (+ 619us) server_negotiation.cc:407] Connection header received
0927 10:57:02.117620 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:02.117624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:02.117676 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:02.117734 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:02.119311 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.120186 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.121854 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.122244 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.122321 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:02.127072 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:02.127075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:02.127078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:02.127119 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:02.129370 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:02.130021 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:02.130025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:02.130027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:02.130124 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:02.130550 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:02.130553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:02.130555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:02.130804 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:02.130887 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:02.131659 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 10:57:02.131794 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1633,"thread_start_us":122,"threads_started":1}
I20260927 10:57:02.139794 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.114672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45789 (local address 127.1.46.193:46217)
0927 10:57:02.116291 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:02.116345 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:02.117386 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:02.118133 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:02.118141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:02.118478 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:02.119141 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:02.119158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.120335 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.120338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:02.121717 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:02.121727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.122526 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.122529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:02.122765 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:02.123467 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:02.123488 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:02.126876 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:02.129509 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:02.129515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:02.129532 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:02.129901 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:02.130225 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:02.130228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:02.130230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:02.130435 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:02.130955 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:02.130961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:02.131287 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 10:57:02.138206 (+ 6919us) client_negotiation.cc:240] Negotiation successful
0927 10:57:02.138658 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":136,"threads_started":1}
I20260927 10:57:02.141396 13645 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45789
I20260927 10:57:02.141631 13645 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:02.142194 13645 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:02.144033 13433 ts_manager.cc:194] Registered new tserver with Master: afd325f3f02f4e3c987e463d3748b0ee (127.1.46.193:38305)
I20260927 10:57:02.145536 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:46217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:57:02.165089 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45789
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 10:57:02.325809 13653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:02.326203 13653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:02.326314 13653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:02.332079 13653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:02.332211 13653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:02.332355 13653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:57:02.339879 13653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.46.212:39755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:02.341282 13653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:02.342553 13653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:02.355099 13660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:02.355099 13662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:02.356276 13659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:02.356576 13653 server_base.cc:1034] running on GCE node
I20260927 10:57:02.357486 13653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:02.358551 13653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:02.374140 13653 hybrid_clock.cc:648] HybridClock initialized: now 1790506622374040 us; error 105 us; skew 500 ppm
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:02.378068 13653 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:57:02.379333 13653 webserver.cc:460] Webserver started at http://127.1.46.194:40255/ using document root <none> and password file <none>
I20260927 10:57:02.380126 13653 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:02.380264 13653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:02.380535 13653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:02.403100 13653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "e452eb357c02402eb162769486c2ecc4"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "d4f6e098325974039d6412c78705fa31"
server_key_iv: "e2e7a85afec4e99d8ce954582f0e3938"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.403975 13653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "e452eb357c02402eb162769486c2ecc4"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "d4f6e098325974039d6412c78705fa31"
server_key_iv: "e2e7a85afec4e99d8ce954582f0e3938"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.412034 13653 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.004s
I20260927 10:57:02.429617 13669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.431351 13653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:02.431469 13653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "e452eb357c02402eb162769486c2ecc4"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "d4f6e098325974039d6412c78705fa31"
server_key_iv: "e2e7a85afec4e99d8ce954582f0e3938"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.431588 13653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:02.474138 13653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:02.477764 13653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:02.477994 13653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:02.478677 13653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:02.479689 13653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:02.479791 13653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.479897 13653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:02.479957 13653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.493044 13653 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:37159
I20260927 10:57:02.494221 13653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:02.495164 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 13653
I20260927 10:57:02.495288 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:02.495669 1211 external_mini_cluster.cc:1428] Setting key fedccab218735e29b74e38edad2fd01b
I20260927 10:57:02.494839 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:37159 every 8 connection(s)
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:57:02.518927 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.498565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:41743 (local address 127.1.46.254:45789)
0927 10:57:02.498722 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:02.498728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:02.499360 (+ 632us) server_negotiation.cc:407] Connection header received
0927 10:57:02.500527 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:02.500531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:02.500579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:02.500629 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:02.502379 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.503208 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.504957 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.505305 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.505383 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:02.509966 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:02.509970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:02.509973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:02.510015 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:02.511833 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:02.512457 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:02.512460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:02.512462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:02.512544 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:02.512931 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:02.512934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:02.512936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:02.513139 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:02.513205 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:02.518628 (+ 5423us) server_negotiation.cc:299] Negotiation successful
0927 10:57:02.518758 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:57:02.520640 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.497574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45789 (local address 127.1.46.194:41743)
0927 10:57:02.499206 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:02.499257 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:02.500275 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:02.501008 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:02.501016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:02.501377 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:02.501988 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:02.502005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.503384 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.503387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:02.504806 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:02.504818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.505569 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.505573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:02.505757 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:02.506456 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:02.506475 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:02.509777 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:02.511983 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:02.511988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:02.512002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:02.512338 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:02.512655 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:02.512658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:02.512660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:02.512821 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:02.513293 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:02.513299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:02.513572 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 10:57:02.519356 (+ 5784us) client_negotiation.cc:240] Negotiation successful
0927 10:57:02.519675 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1407,"thread_start_us":124,"threads_started":1}
I20260927 10:57:02.522105 13783 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45789
I20260927 10:57:02.522346 13783 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:02.522985 13783 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:02.524048 13433 ts_manager.cc:194] Registered new tserver with Master: e452eb357c02402eb162769486c2ecc4 (127.1.46.194:37159)
I20260927 10:57:02.524771 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:41743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:57:02.545372 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45789
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:02.697098 13788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:02.697487 13788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:02.697593 13788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:02.703913 13788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:02.704051 13788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:02.704185 13788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:57:02.711783 13788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.46.212:39755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:02.713145 13788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:02.714476 13788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
W20260927 10:57:02.728108 13794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:02.728812 13793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:02.729599 13788 server_base.cc:1034] running on GCE node
W20260927 10:57:02.729879 13796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:02.730317 13788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:02.731014 13788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:02.732191 13788 hybrid_clock.cc:648] HybridClock initialized: now 1790506622732132 us; error 72 us; skew 500 ppm
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:02.735647 13788 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:57:02.736557 13788 webserver.cc:460] Webserver started at http://127.1.46.195:34865/ using document root <none> and password file <none>
I20260927 10:57:02.737205 13788 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:02.737275 13788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:02.737480 13788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:02.756723 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "5147b31e43d8480098af73a598ab424f"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "3ce3f0f65e9ed42e4733935d584be36a"
server_key_iv: "2cc65ac3b418e266eefe81b470b3d5af"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.761335 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "5147b31e43d8480098af73a598ab424f"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "3ce3f0f65e9ed42e4733935d584be36a"
server_key_iv: "2cc65ac3b418e266eefe81b470b3d5af"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.765949 13788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:57:02.801162 13803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.802927 13788 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.003s sys 0.000s
I20260927 10:57:02.803049 13788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "5147b31e43d8480098af73a598ab424f"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "3ce3f0f65e9ed42e4733935d584be36a"
server_key_iv: "2cc65ac3b418e266eefe81b470b3d5af"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:02.803193 13788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:02.828387 13788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:02.832248 13788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:02.832504 13788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:02.833271 13788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:02.834304 13788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:02.834404 13788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.834498 13788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:02.834570 13788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:02.844681 13788 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:46499
I20260927 10:57:02.845770 13788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:02.848479 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 13788
I20260927 10:57:02.848614 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:02.848959 1211 external_mini_cluster.cc:1428] Setting key 16c9dadc74b4fe046d19b9777261c940
I20260927 10:57:02.852370 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:46499 every 8 connection(s)
Sep 27 10:57:02 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:02.868351 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.854040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:39997 (local address 127.1.46.254:45789)
0927 10:57:02.854195 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:02.854199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:02.855752 (+ 1553us) server_negotiation.cc:407] Connection header received
0927 10:57:02.856961 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:02.856965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:02.857013 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:02.857078 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:02.858677 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.859498 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.861127 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.861474 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.861544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:02.864416 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:02.864420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:02.864422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:02.864448 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:02.866143 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:02.866719 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:02.866722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:02.866724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:02.866801 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:02.867157 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:02.867160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:02.867161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:02.867353 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:02.867426 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:02.868033 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 10:57:02.868180 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:57:02.869949 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:02.854525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45789 (local address 127.1.46.195:39997)
0927 10:57:02.855615 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:02.855665 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:02.856714 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:02.857485 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:02.857493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:02.857834 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:02.858495 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:02.858512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.859641 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.859644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:02.860987 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:02.860997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:02.861740 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:02.861743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:02.861920 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:02.862596 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:02.862617 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:02.864248 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:02.866272 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:02.866277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:02.866292 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:02.866611 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:02.866895 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:02.866898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:02.866900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:02.867048 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:02.867485 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:02.867491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:02.867757 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 10:57:02.868718 (+ 961us) client_negotiation.cc:240] Negotiation successful
0927 10:57:02.869012 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":129,"threads_started":1}
I20260927 10:57:02.871349 13917 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45789
I20260927 10:57:02.871562 13917 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:02.872048 13917 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:02.873063 13433 ts_manager.cc:194] Registered new tserver with Master: 5147b31e43d8480098af73a598ab424f (127.1.46.195:46499)
I20260927 10:57:02.873734 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:39997
I20260927 10:57:02.874586 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:02.875276 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 13514
I20260927 10:57:02.883972 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 13653
I20260927 10:57:02.890527 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 13788
I20260927 10:57:02.896274 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 13401
I20260927 10:57:02.903644 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45789
--webserver_interface=127.1.46.254
--webserver_port=33675
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45789
--ranger_config_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:03.033380 13920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:03.033634 13920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:03.033727 13920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:03.037638 13920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:03.037711 13920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:03.037748 13920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:03.037806 13920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:03.037845 13920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:03.043061 13920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45789
--ranger_config_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:03.043936 13920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:03.044874 13920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:03.053756 13928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.053789 13925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.054095 13926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:03.054452 13920 server_base.cc:1034] running on GCE node
I20260927 10:57:03.054984 13920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:03.055989 13920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:03.057164 13920 hybrid_clock.cc:648] HybridClock initialized: now 1790506623057155 us; error 27 us; skew 500 ppm
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:03.059827 13920 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:03.060700 13920 webserver.cc:460] Webserver started at http://127.1.46.254:33675/ using document root <none> and password file <none>
I20260927 10:57:03.061156 13920 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:03.061216 13920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:03.136049 13920 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.000s
I20260927 10:57:03.138689 13936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.140197 13920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:57:03.140331 13920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "075b114aa3a940948fd5c0fa02b5f578"
format_stamp: "Formatted at 2026-09-27 10:57:01 on dist-test-slave-cffh"
server_key: "df7c75532e1b9ee2ee8bcfad035d0db1"
server_key_iv: "09ae16381f3fed97193f6f43467dbda8"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:03.141016 13920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:03.169840 13920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:03.173103 13920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:03.173309 13920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:03.180962 13920 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45789
I20260927 10:57:03.180994 13988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45789 every 8 connection(s)
I20260927 10:57:03.182204 13920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:03.186383 13989 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:57:03.188047 13989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Bootstrap starting.
I20260927 10:57:03.191851 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 13920
I20260927 10:57:03.192345 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:38305
--local_ip_for_outbound_sockets=127.1.46.193
--tserver_master_addrs=127.1.46.254:45789
--webserver_port=35117
--webserver_interface=127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:03.204967 13989 log.cc:824] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:03.211354 13989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:57:03.211750 13989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Bootstrap complete.
I20260927 10:57:03.215842 13989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:03.216249 13989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075b114aa3a940948fd5c0fa02b5f578, State: Initialized, Role: FOLLOWER
I20260927 10:57:03.216773 13989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:03.216907 13989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:03.216984 13989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:03.217103 13989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:57:03.219029 13989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:03.219372 13989 leader_election.cc:302] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 075b114aa3a940948fd5c0fa02b5f578; no voters:
I20260927 10:57:03.219681 13989 leader_election.cc:288] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:57:03.219770 13993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:57:03.220026 13993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [term 2 LEADER]: Becoming Leader. State: Replica: 075b114aa3a940948fd5c0fa02b5f578, State: Running, Role: LEADER
I20260927 10:57:03.220345 13993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } }
I20260927 10:57:03.220918 13989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:03.221303 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "075b114aa3a940948fd5c0fa02b5f578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } } }
I20260927 10:57:03.221475 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:03.221731 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 075b114aa3a940948fd5c0fa02b5f578. Latest consensus state: current_term: 2 leader_uuid: "075b114aa3a940948fd5c0fa02b5f578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b114aa3a940948fd5c0fa02b5f578" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45789 } } }
I20260927 10:57:03.221885 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:03.222177 13998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:03.225471 13998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:03.225883 13989 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:57:03.226233 13998 catalog_manager.cc:1259] Loaded cluster ID: d23e20eb7aca49d1b5dcd181373b49a8
I20260927 10:57:03.226343 13998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:03.228760 13998 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:03.229454 13998 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Loaded TSK: 0
I20260927 10:57:03.230710 13998 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:57:03.340700 13991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:03.341362 13991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:03.341492 13991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:03.346967 13991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:03.347096 13991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:03.347210 13991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:57:03.354804 13991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:38305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=35117
--tserver_master_addrs=127.1.46.254:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:03.357158 13991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:03.358409 13991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:03.369904 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.370409 14026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.370710 14029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:03.371935 13991 server_base.cc:1034] running on GCE node
I20260927 10:57:03.372342 13991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:03.373029 13991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:03.374199 13991 hybrid_clock.cc:648] HybridClock initialized: now 1790506623374136 us; error 77 us; skew 500 ppm
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:03.377676 13991 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:57:03.378536 13991 webserver.cc:460] Webserver started at http://127.1.46.193:35117/ using document root <none> and password file <none>
I20260927 10:57:03.379164 13991 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:03.379237 13991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:03.408013 13991 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:57:03.410820 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.412230 13991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:57:03.412334 13991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "afd325f3f02f4e3c987e463d3748b0ee"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "49c9163a9a56febfb9e9bb5bdab9acf8"
server_key_iv: "b589faf0b6d1ed55d0341458965fe1cf"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:03.412950 13991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:03.430171 13991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:03.433543 13991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:03.433768 13991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:03.434396 13991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:03.435390 13991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:03.435483 13991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.435596 13991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:03.435679 13991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.447431 13991 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:38305
I20260927 10:57:03.449263 13991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:03.449985 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 13991
I20260927 10:57:03.450609 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:37159
--local_ip_for_outbound_sockets=127.1.46.194
--tserver_master_addrs=127.1.46.254:45789
--webserver_port=40255
--webserver_interface=127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:03.450749 14150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:38305 every 8 connection(s)
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:03.486044 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:03.453145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45789 (local address 127.1.46.193:57877)
0927 10:57:03.454282 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:03.454336 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:03.455398 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:03.471996 (+ 16598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:03.472005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:03.472342 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:03.472989 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:03.473006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.475132 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.475136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:03.476518 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:03.476529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.477563 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.477566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:03.477758 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:03.478458 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:03.478477 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:03.480186 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:03.482695 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:03.482700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:03.482714 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:03.483036 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:03.483428 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:03.483430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:03.483433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:03.483588 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:03.484294 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:03.484301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:03.484581 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 10:57:03.484776 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 10:57:03.485078 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":125,"threads_started":1}
I20260927 10:57:03.488055 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:03.453077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:57877 (local address 127.1.46.254:45789)
0927 10:57:03.462766 (+ 9689us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:03.462788 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:03.462837 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:57:03.463158 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:03.463174 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:03.463614 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:03.473234 (+ 9620us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:03.473552 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.474977 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.476681 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.477389 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.478125 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:03.480408 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:03.480415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:03.480433 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:03.480462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:03.482515 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:03.483180 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:03.483186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:03.483191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:03.483319 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:03.483717 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:03.483720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:03.483721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:03.484168 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:03.486756 (+ 2588us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:03.486894 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 10:57:03.487155 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9501,"thread_start_us":133,"threads_started":1}
I20260927 10:57:03.490828 14151 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45789
I20260927 10:57:03.491056 14151 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:03.491593 14151 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
W20260927 10:57:03.493956 14151 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:45789 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:57:03.494076 14151 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:03.494254 14151 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:03.495241 14151 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:03.495409 14151 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
W20260927 10:57:03.496217 14151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:57:03.618546 14153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:03.619088 14153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:03.619300 14153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:03.625824 14153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:03.625943 14153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:03.626097 14153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:57:03.633900 14153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:37159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=40255
--tserver_master_addrs=127.1.46.254:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:03.635208 14153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:03.636543 14153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:57:03.649981 14153 server_base.cc:1034] running on GCE node
W20260927 10:57:03.649683 14161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.650823 14163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.651141 14160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:03.651722 14153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:03.652482 14153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:03.653682 14153 hybrid_clock.cc:648] HybridClock initialized: now 1790506623653646 us; error 53 us; skew 500 ppm
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:03.657488 14153 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:57:03.658412 14153 webserver.cc:460] Webserver started at http://127.1.46.194:40255/ using document root <none> and password file <none>
I20260927 10:57:03.659097 14153 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:03.659205 14153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:03.681166 14153 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:57:03.684137 14171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.685698 14153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:03.685863 14153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "e452eb357c02402eb162769486c2ecc4"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "d4f6e098325974039d6412c78705fa31"
server_key_iv: "e2e7a85afec4e99d8ce954582f0e3938"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:03.686432 14153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:03.706825 14153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:03.709805 14153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:03.710001 14153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:03.710633 14153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:03.711473 14153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:03.711539 14153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.711648 14153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:03.711691 14153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.721585 14153 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:37159
I20260927 10:57:03.721611 14284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:37159 every 8 connection(s)
I20260927 10:57:03.723587 14153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:03.727288 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 14153
I20260927 10:57:03.727783 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.46.212:39755/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:46499
--local_ip_for_outbound_sockets=127.1.46.195
--tserver_master_addrs=127.1.46.254:45789
--webserver_port=34865
--webserver_interface=127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:03.741026 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:03.725784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:45391 (local address 127.1.46.254:45789)
0927 10:57:03.725901 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:03.725906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:03.726762 (+ 856us) server_negotiation.cc:407] Connection header received
0927 10:57:03.728365 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:03.728372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:03.728435 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:03.728491 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:03.729972 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.730877 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.732159 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.732430 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.732545 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:03.736262 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:03.736269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:03.736274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:03.736304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:03.738234 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:03.738981 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:03.738987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:03.738991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:03.739086 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:03.739530 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:03.739534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:03.739539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:03.739814 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:03.739912 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:03.740629 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 10:57:03.740789 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:57:03.742695 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:03.726144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45789 (local address 127.1.46.194:45391)
0927 10:57:03.726618 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:03.726668 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:03.728104 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:03.728890 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:03.728901 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:03.729274 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:03.729830 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:03.729843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.731038 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.731044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:03.732031 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:03.732040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:03.732579 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:03.732583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:03.732814 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:03.733445 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:03.733470 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:03.736046 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:03.738400 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:03.738409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:03.738426 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:03.738846 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:03.739213 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:03.739218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:03.739222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:03.739407 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:03.739988 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:03.739998 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:03.740322 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 10:57:03.741201 (+ 879us) client_negotiation.cc:240] Negotiation successful
0927 10:57:03.741549 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":144,"threads_started":1}
I20260927 10:57:03.744086 14285 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45789
I20260927 10:57:03.744320 14285 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:03.744798 14285 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
W20260927 10:57:03.746249 14285 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:45789 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:57:03.746357 14285 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:03.746517 14285 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:03.747360 14285 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:03.747509 14285 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
W20260927 10:57:03.748209 14285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:57:03.888432 14288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:03.888803 14288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:03.888875 14288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:03.895498 14288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:03.895570 14288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:03.895656 14288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:57:03.903283 14288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.46.212:39755/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:46499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=34865
--tserver_master_addrs=127.1.46.254:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:03.904584 14288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:03.905880 14288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:03.918301 14294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.918357 14295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:03.918749 14297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:03.919195 14288 server_base.cc:1034] running on GCE node
I20260927 10:57:03.919626 14288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:03.920258 14288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:03.921450 14288 hybrid_clock.cc:648] HybridClock initialized: now 1790506623921413 us; error 51 us; skew 500 ppm
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:03.925242 14288 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:57:03.926149 14288 webserver.cc:460] Webserver started at http://127.1.46.195:34865/ using document root <none> and password file <none>
I20260927 10:57:03.926795 14288 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:03.926905 14288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:57:03 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:03.955077 14288 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:03.962779 14305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.964440 14288 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260927 10:57:03.964570 14288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "5147b31e43d8480098af73a598ab424f"
format_stamp: "Formatted at 2026-09-27 10:57:02 on dist-test-slave-cffh"
server_key: "3ce3f0f65e9ed42e4733935d584be36a"
server_key_iv: "2cc65ac3b418e266eefe81b470b3d5af"
server_key_version: "kuduclusterkey@0"
I20260927 10:57:03.965313 14288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:03.986442 14288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:03.990031 14288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:03.990231 14288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:03.990969 14288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:03.991972 14288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:03.992027 14288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:03.992079 14288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:03.992101 14288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:04.004997 14288 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:46499
I20260927 10:57:04.005152 14418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:46499 every 8 connection(s)
I20260927 10:57:04.006230 14288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:04.014360 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 14288
I20260927 10:57:04.026096 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:04.015554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.1.46.254:45789)
0927 10:57:04.015701 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:04.015706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:04.015983 (+ 277us) server_negotiation.cc:407] Connection header received
0927 10:57:04.016212 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:04.016216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:04.016264 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:04.016318 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:04.017443 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:04.018203 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:04.019542 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:04.019874 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:04.019962 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:04.021935 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:04.021939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:04.021942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:04.021966 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:04.023806 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:04.024303 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:04.024306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:04.024308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:04.024387 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:04.024723 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:04.024726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:04.024728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:04.024919 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:04.025090 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:04.025782 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 10:57:04.025895 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 10:57:04 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:04.032521 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:04.017914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:50675 (local address 127.1.46.254:45789)
0927 10:57:04.019794 (+ 1880us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:04.019804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:04.019815 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:57:04.019847 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:04.019849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:04.019891 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:04.019967 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:04.021547 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:04.022326 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:04.024142 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:04.024501 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:04.024578 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:04.028508 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:04.028512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:04.028514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:04.028538 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:04.030321 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:04.030943 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:04.030947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:04.030950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:04.031037 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:04.031445 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:04.031449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:04.031452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:04.031627 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:04.031735 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:04.032230 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 10:57:04.032346 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1808,"thread_start_us":84,"threads_started":1}
I20260927 10:57:04.033380 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:04.016814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45789 (local address 127.1.46.195:50675)
0927 10:57:04.018334 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:04.018392 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:04.019405 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:04.020253 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:04.020262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:04.020623 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:04.021361 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:04.021384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:04.022477 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:04.022483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:04.024007 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:04.024018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:04.024762 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:04.024769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:04.024975 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:04.025562 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:04.025582 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:04.028331 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:04.030451 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:04.030457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:04.030473 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:04.030847 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:04.031137 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:04.031141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:04.031143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:04.031314 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:04.031744 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:04.031753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:04.031991 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:57:04.032183 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 10:57:04.032486 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"thread_start_us":160,"threads_started":1}
I20260927 10:57:04.034499 14419 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45789
I20260927 10:57:04.034759 14419 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:04.035259 14419 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
W20260927 10:57:04.036763 14419 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:45789 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:57:04.036881 14419 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:04.037060 14419 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:04.037907 14419 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:04.038069 14419 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
W20260927 10:57:04.039036 14419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:57:04 dist-test-slave-cffh krb5kdc[12957](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:04.496450 14151 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:04.496781 14151 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:04.497979 13953 ts_manager.cc:194] Registered new tserver with Master: afd325f3f02f4e3c987e463d3748b0ee (127.1.46.193:38305)
I20260927 10:57:04.499711 13953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:57877
I20260927 10:57:04.748407 14285 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:04.748692 14285 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:04.749470 13953 ts_manager.cc:194] Registered new tserver with Master: e452eb357c02402eb162769486c2ecc4 (127.1.46.194:37159)
I20260927 10:57:04.750145 13953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:45391
Sep 27 10:57:04 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:05.039250 14419 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:05.039548 14419 heartbeater.cc:511] Master 127.1.46.254:45789 requested a full tablet report, sending...
I20260927 10:57:05.040308 13953 ts_manager.cc:194] Registered new tserver with Master: 5147b31e43d8480098af73a598ab424f (127.1.46.195:46499)
I20260927 10:57:05.041100 13953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:50675
I20260927 10:57:05.049432 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:05.064566 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.051050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.1.46.254:45789)
0927 10:57:05.051421 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.051426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.051599 (+ 173us) server_negotiation.cc:407] Connection header received
0927 10:57:05.051712 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.051715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.051767 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.051862 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:05.052776 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.053548 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.054784 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.055112 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.055212 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.060781 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:05.060786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:05.060788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:05.060815 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:05.062653 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:05.063156 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:05.063159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:05.063162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:05.063245 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:05.063560 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:05.063563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:05.063565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:05.063777 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:05.063923 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.064165 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.064276 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":78,"threads_started":1}
I20260927 10:57:05.068318 13953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:05.072799 13953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:05.094233 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.087785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:46499 (local address 127.0.0.1:49684)
0927 10:57:05.088712 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.088725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:05.088852 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:05.089556 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:05.089562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:05.089583 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:05.089700 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.089706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.091143 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.091147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.092455 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.092466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.093726 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.093729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.093830 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.093866 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:57:05.093971 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:57:05.094082 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":73,"threads_started":1}
I20260927 10:57:05.095065 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.088179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.1.46.195:46499)
0927 10:57:05.088606 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.088613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.088984 (+ 371us) server_negotiation.cc:407] Connection header received
0927 10:57:05.089060 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.089065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.089262 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.089406 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:05.089817 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.091022 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.092610 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.093429 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.093530 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.093669 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.094831 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.094942 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":95,"threads_started":1}
I20260927 10:57:05.098274 14353 tablet_service.cc:1467] Processing CreateTablet for tablet bb02765efbef43a3812b6e1705744666 (DEFAULT_TABLE table=test-table [id=6f4ba816b0304e5ebf626c5eb1e4ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:05.099476 14353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb02765efbef43a3812b6e1705744666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:05.100478 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.086193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:38305 (local address 127.0.0.1:33886)
0927 10:57:05.088633 (+ 2440us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.088683 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:05.088855 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:05.090234 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:05.090236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:05.090253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:05.090339 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.090345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.095564 (+ 5219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.095567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.096764 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.096775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.100078 (+ 3303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.100080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.100144 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.100160 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:57:05.100269 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:57:05.100340 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2262,"thread_start_us":90,"threads_started":1}
I20260927 10:57:05.100533 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.086998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.1.46.193:38305)
0927 10:57:05.087342 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.087349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.088776 (+ 1427us) server_negotiation.cc:407] Connection header received
0927 10:57:05.089742 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.089749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.089925 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.090037 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:05.090472 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.092469 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.098295 (+ 5826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.099827 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.099897 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.100001 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.100302 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.100400 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":83,"threads_started":1}
I20260927 10:57:05.103431 14085 tablet_service.cc:1467] Processing CreateTablet for tablet bb02765efbef43a3812b6e1705744666 (DEFAULT_TABLE table=test-table [id=6f4ba816b0304e5ebf626c5eb1e4ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:05.104522 14085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb02765efbef43a3812b6e1705744666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:05.107661 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.088058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:37159 (local address 127.0.0.1:56336)
0927 10:57:05.092760 (+ 4702us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.092774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:05.092855 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:05.103410 (+ 10555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:05.103414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:05.103435 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:05.103521 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.103526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.104909 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.104913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.106008 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.106019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.107275 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.107278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.107360 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.107385 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:05.107458 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:57:05.107527 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4642,"thread_start_us":97,"threads_started":1}
I20260927 10:57:05.108197 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.088182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.1.46.194:37159)
0927 10:57:05.102778 (+ 14596us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.102785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.102810 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:57:05.102894 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.102899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.103111 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.103260 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:05.103657 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.104769 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.106147 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.106971 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.107075 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.107212 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.107965 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.108066 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14479,"thread_start_us":85,"threads_started":1}
I20260927 10:57:05.109696 14453 tablet_bootstrap.cc:492] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: Bootstrap starting.
I20260927 10:57:05.111130 14200 tablet_service.cc:1467] Processing CreateTablet for tablet bb02765efbef43a3812b6e1705744666 (DEFAULT_TABLE table=test-table [id=6f4ba816b0304e5ebf626c5eb1e4ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:05.111967 14453 tablet_bootstrap.cc:654] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:05.112118 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb02765efbef43a3812b6e1705744666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:05.112811 14453 log.cc:824] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:05.114831 14453 tablet_bootstrap.cc:492] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: No bootstrap required, opened a new log
I20260927 10:57:05.115080 14453 ts_tablet_manager.cc:1397] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 10:57:05.120293 14456 tablet_bootstrap.cc:492] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Bootstrap starting.
I20260927 10:57:05.122450 14456 tablet_bootstrap.cc:654] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:05.123363 14456 log.cc:824] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:05.125396 14456 tablet_bootstrap.cc:492] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: No bootstrap required, opened a new log
I20260927 10:57:05.125587 14456 ts_tablet_manager.cc:1397] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:57:05.129810 14456 raft_consensus.cc:348] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.130076 14456 raft_consensus.cc:374] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:05.130124 14456 raft_consensus.cc:729] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e452eb357c02402eb162769486c2ecc4, State: Initialized, Role: FOLLOWER
Sep 27 10:57:05 dist-test-slave-cffh krb5kdc[12957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for HTTP/127.1.46.212@KRBTEST.COM
I20260927 10:57:05.130864 14456 consensus_queue.cc:260] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.132032 14456 ts_tablet_manager.cc:1428] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:57:05.132413 14285 heartbeater.cc:503] Master 127.1.46.254:45789 was elected leader, sending a full tablet report...
I20260927 10:57:05.138924 14453 raft_consensus.cc:348] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.139290 14453 raft_consensus.cc:374] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:05.139402 14453 raft_consensus.cc:729] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5147b31e43d8480098af73a598ab424f, State: Initialized, Role: FOLLOWER
I20260927 10:57:05.145848 14453 consensus_queue.cc:260] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.147809 14419 heartbeater.cc:503] Master 127.1.46.254:45789 was elected leader, sending a full tablet report...
I20260927 10:57:05.149843 14453 ts_tablet_manager.cc:1428] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260927 10:57:05.150522 14454 tablet_bootstrap.cc:492] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee: Bootstrap starting.
I20260927 10:57:05.153079 14454 tablet_bootstrap.cc:654] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:05.153965 14454 log.cc:824] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:05.158138 14454 tablet_bootstrap.cc:492] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee: No bootstrap required, opened a new log
I20260927 10:57:05.158402 14454 ts_tablet_manager.cc:1397] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 10:57:05.169340 14454 raft_consensus.cc:348] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.170306 14454 raft_consensus.cc:374] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:05.170403 14454 raft_consensus.cc:729] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd325f3f02f4e3c987e463d3748b0ee, State: Initialized, Role: FOLLOWER
I20260927 10:57:05.174769 14454 consensus_queue.cc:260] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.177385 14454 ts_tablet_manager.cc:1428] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260927 10:57:05.178655 14151 heartbeater.cc:503] Master 127.1.46.254:45789 was elected leader, sending a full tablet report...
W20260927 10:57:05.202667 14152 tablet.cc:2369] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:05.212219 14461 raft_consensus.cc:482] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:05.212431 14461 raft_consensus.cc:504] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.213775 14461 leader_election.cc:288] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5147b31e43d8480098af73a598ab424f (127.1.46.195:46499), e452eb357c02402eb162769486c2ecc4 (127.1.46.194:37159)
I20260927 10:57:05.219771 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.214492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:56345 (local address 127.1.46.195:46499)
0927 10:57:05.214641 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.214645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.214659 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:57:05.215609 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.215613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.215670 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.215755 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:05.216098 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.216823 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.218460 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.219094 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.219184 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.219235 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.219571 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.219643 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:57:05.220494 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.214091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:46499 (local address 127.1.46.193:56345)
0927 10:57:05.214414 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.214430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:05.215450 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:05.215785 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:05.215788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:05.215809 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:05.215959 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.215965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.216970 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.216974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.218317 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.218328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.219327 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.219330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.219439 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.219461 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:57:05.220155 (+ 694us) client_negotiation.cc:240] Negotiation successful
0927 10:57:05.220236 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260927 10:57:05.221148 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb02765efbef43a3812b6e1705744666" candidate_uuid: "afd325f3f02f4e3c987e463d3748b0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5147b31e43d8480098af73a598ab424f" is_pre_election: true
I20260927 10:57:05.221514 14361 raft_consensus.cc:2454] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd325f3f02f4e3c987e463d3748b0ee in term 0.
I20260927 10:57:05.223529 14040 leader_election.cc:302] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5147b31e43d8480098af73a598ab424f, afd325f3f02f4e3c987e463d3748b0ee; no voters:
I20260927 10:57:05.223902 14461 raft_consensus.cc:2781] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:05.223980 14461 raft_consensus.cc:482] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:05.224030 14461 raft_consensus.cc:3037] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:05.223987 14458 raft_consensus.cc:482] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:05.224154 14458 raft_consensus.cc:504] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.225289 14461 raft_consensus.cc:504] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.225359 14458 leader_election.cc:288] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afd325f3f02f4e3c987e463d3748b0ee (127.1.46.193:38305), 5147b31e43d8480098af73a598ab424f (127.1.46.195:46499)
I20260927 10:57:05.225600 14461 leader_election.cc:288] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [CANDIDATE]: Term 1 election: Requested vote from peers 5147b31e43d8480098af73a598ab424f (127.1.46.195:46499), e452eb357c02402eb162769486c2ecc4 (127.1.46.194:37159)
I20260927 10:57:05.226166 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb02765efbef43a3812b6e1705744666" candidate_uuid: "afd325f3f02f4e3c987e463d3748b0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5147b31e43d8480098af73a598ab424f"
I20260927 10:57:05.226310 14361 raft_consensus.cc:3037] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:57:05.226745 14286 tablet.cc:2369] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:05.228147 14361 raft_consensus.cc:2454] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd325f3f02f4e3c987e463d3748b0ee in term 1.
I20260927 10:57:05.228207 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.215296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:34163 (local address 127.1.46.194:37159)
0927 10:57:05.215404 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.215409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.221012 (+ 5603us) server_negotiation.cc:407] Connection header received
0927 10:57:05.221194 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.221197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.221250 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.221298 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:05.221642 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.222413 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.227033 (+ 4620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.227602 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.227676 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.227719 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.228039 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.228103 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:57:05.228570 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.214923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:37159 (local address 127.1.46.193:34163)
0927 10:57:05.220932 (+ 6009us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.220947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:05.221080 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:05.221425 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:05.221427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:05.221443 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:05.221525 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.221530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.222549 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.222552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.226910 (+ 4358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.226920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.227800 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.227803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.227897 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.227933 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:57:05.228362 (+ 429us) client_negotiation.cc:240] Negotiation successful
0927 10:57:05.228451 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5940,"thread_start_us":93,"threads_started":1}
I20260927 10:57:05.230075 14222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb02765efbef43a3812b6e1705744666" candidate_uuid: "afd325f3f02f4e3c987e463d3748b0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e452eb357c02402eb162769486c2ecc4" is_pre_election: true
I20260927 10:57:05.230279 14222 raft_consensus.cc:2454] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd325f3f02f4e3c987e463d3748b0ee in term 0.
I20260927 10:57:05.230549 14223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb02765efbef43a3812b6e1705744666" candidate_uuid: "afd325f3f02f4e3c987e463d3748b0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e452eb357c02402eb162769486c2ecc4"
I20260927 10:57:05.230664 14223 raft_consensus.cc:3037] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:05.231783 14223 raft_consensus.cc:2454] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd325f3f02f4e3c987e463d3748b0ee in term 1.
I20260927 10:57:05.228582 14040 leader_election.cc:302] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5147b31e43d8480098af73a598ab424f, afd325f3f02f4e3c987e463d3748b0ee; no voters:
I20260927 10:57:05.232730 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.226638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:39063 (local address 127.1.46.195:46499)
0927 10:57:05.226740 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.226744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.228912 (+ 2168us) server_negotiation.cc:407] Connection header received
0927 10:57:05.229066 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.229068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.229106 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.229147 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:05.229477 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.230106 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.231585 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.232109 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.232191 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.232230 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.232550 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.232623 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:57:05.233373 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.226056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:46499 (local address 127.1.46.194:39063)
0927 10:57:05.228820 (+ 2764us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.228834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:05.228971 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:05.229251 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:05.229254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:05.229267 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:05.229366 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.229370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.230249 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.230252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.231438 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.231450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.232317 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.232320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.232412 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.232433 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:57:05.233170 (+ 737us) client_negotiation.cc:240] Negotiation successful
0927 10:57:05.233242 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2699,"thread_start_us":90,"threads_started":1}
I20260927 10:57:05.233946 14361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb02765efbef43a3812b6e1705744666" candidate_uuid: "e452eb357c02402eb162769486c2ecc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5147b31e43d8480098af73a598ab424f" is_pre_election: true
I20260927 10:57:05.234110 14361 raft_consensus.cc:2379] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e452eb357c02402eb162769486c2ecc4 in current term 1: Already voted for candidate afd325f3f02f4e3c987e463d3748b0ee in this term.
I20260927 10:57:05.238696 14461 raft_consensus.cc:2781] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:05.238976 14461 raft_consensus.cc:686] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [term 1 LEADER]: Becoming Leader. State: Replica: afd325f3f02f4e3c987e463d3748b0ee, State: Running, Role: LEADER
I20260927 10:57:05.239359 14461 consensus_queue.cc:237] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } }
I20260927 10:57:05.239897 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.226374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:34943 (local address 127.1.46.193:38305)
0927 10:57:05.226480 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.226485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.226897 (+ 412us) server_negotiation.cc:407] Connection header received
0927 10:57:05.227065 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.227067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.227105 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.227147 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:05.227508 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.228156 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.238690 (+ 10534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.239258 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.239344 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.239405 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.239732 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.239799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:57:05.240437 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.225664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:38305 (local address 127.1.46.194:34943)
0927 10:57:05.226818 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.226831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:05.226962 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:05.227251 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:05.227254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:05.227269 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:05.227398 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.227404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.228277 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.228280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.229469 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:05.229476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.239505 (+ 10029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.239509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:05.239602 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.239623 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:57:05.240183 (+ 560us) client_negotiation.cc:240] Negotiation successful
0927 10:57:05.240292 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"thread_start_us":126,"threads_started":1}
I20260927 10:57:05.241010 14102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb02765efbef43a3812b6e1705744666" candidate_uuid: "e452eb357c02402eb162769486c2ecc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd325f3f02f4e3c987e463d3748b0ee" is_pre_election: true
I20260927 10:57:05.242477 14174 leader_election.cc:302] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e452eb357c02402eb162769486c2ecc4; no voters: 5147b31e43d8480098af73a598ab424f, afd325f3f02f4e3c987e463d3748b0ee
I20260927 10:57:05.242967 14458 raft_consensus.cc:2726] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 10:57:05.242973 13953 catalog_manager.cc:5495] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee reported cstate change: term changed from 0 to 1, leader changed from <none> to afd325f3f02f4e3c987e463d3748b0ee (127.1.46.193). New cstate: current_term: 1 leader_uuid: "afd325f3f02f4e3c987e463d3748b0ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afd325f3f02f4e3c987e463d3748b0ee" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 38305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 } health_report { overall_health: UNKNOWN } } }
W20260927 10:57:05.258766 14420 tablet.cc:2369] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:05.283108 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.279164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.1.46.193:38305)
0927 10:57:05.279299 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:05.279303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:05.279313 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:57:05.279382 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:05.279384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:05.279434 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:05.279560 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:05.279792 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.280414 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.281239 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:05.281496 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:05.281592 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:05.282016 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:05.282568 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:05.282732 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:05.282886 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 10:57:05.282939 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:57:05.290153 14223 raft_consensus.cc:1261] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 1 FOLLOWER]: Refusing update from remote peer afd325f3f02f4e3c987e463d3748b0ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:05.290263 14361 raft_consensus.cc:1261] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 1 FOLLOWER]: Refusing update from remote peer afd325f3f02f4e3c987e463d3748b0ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:05.290879 14467 consensus_queue.cc:1035] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e452eb357c02402eb162769486c2ecc4" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:05.291113 14461 consensus_queue.cc:1035] T bb02765efbef43a3812b6e1705744666 P afd325f3f02f4e3c987e463d3748b0ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "5147b31e43d8480098af73a598ab424f" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 46499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:05.301317 14470 mvcc.cc:205] Tried to move back new op lower bound from 7333915137180815360 to 7333915136983883776. Current Snapshot: MvccSnapshot[applied={T|T < 7333915137180815360}]
I20260927 10:57:05.304015 14471 mvcc.cc:205] Tried to move back new op lower bound from 7333915137180815360 to 7333915136983883776. Current Snapshot: MvccSnapshot[applied={T|T < 7333915137180815360}]
I20260927 10:57:05.310513 14472 mvcc.cc:205] Tried to move back new op lower bound from 7333915137180815360 to 7333915136983883776. Current Snapshot: MvccSnapshot[applied={T|T < 7333915137180815360}]
I20260927 10:57:05.317248 13953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:05.317451 13953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:05.320063 13953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 075b114aa3a940948fd5c0fa02b5f578: Sending DeleteTablet for 3 replicas of tablet bb02765efbef43a3812b6e1705744666
I20260927 10:57:05.321702 14085 tablet_service.cc:1514] Processing DeleteTablet for tablet bb02765efbef43a3812b6e1705744666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:05 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:33886
I20260927 10:57:05.322243 14353 tablet_service.cc:1514] Processing DeleteTablet for tablet bb02765efbef43a3812b6e1705744666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:05 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:49684
I20260927 10:57:05.322312 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 13991
I20260927 10:57:05.322726 14200 tablet_service.cc:1514] Processing DeleteTablet for tablet bb02765efbef43a3812b6e1705744666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:05 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:56336
I20260927 10:57:05.324726 14480 tablet_replica.cc:330] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: stopping tablet replica
I20260927 10:57:05.325026 14480 raft_consensus.cc:2229] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:05.325376 14480 raft_consensus.cc:2258] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:05.326761 14481 tablet_replica.cc:330] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: stopping tablet replica
I20260927 10:57:05.327071 14481 raft_consensus.cc:2229] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:05.327373 14481 raft_consensus.cc:2258] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:05.328892 14481 ts_tablet_manager.cc:1905] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:05.331705 14481 ts_tablet_manager.cc:1918] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:05.331825 14481 log.cc:1196] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/bb02765efbef43a3812b6e1705744666
I20260927 10:57:05.332165 14481 ts_tablet_manager.cc:1939] T bb02765efbef43a3812b6e1705744666 P e452eb357c02402eb162769486c2ecc4: Deleting consensus metadata
W20260927 10:57:05.332870 13939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:38305 (error 108)
I20260927 10:57:05.333480 13940 catalog_manager.cc:4841] TS e452eb357c02402eb162769486c2ecc4 (127.1.46.194:37159): tablet bb02765efbef43a3812b6e1705744666 (table test-table [id=6f4ba816b0304e5ebf626c5eb1e4ffb1]) successfully deleted
I20260927 10:57:05.334134 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 14153
I20260927 10:57:05.340910 14480 ts_tablet_manager.cc:1905] T bb02765efbef43a3812b6e1705744666 P 5147b31e43d8480098af73a598ab424f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:05.342772 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.342331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:38305 (local address 127.0.0.1:33904)
0927 10:57:05.342529 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.342633 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260927 10:57:05.342938 13939 catalog_manager.cc:4584] TS afd325f3f02f4e3c987e463d3748b0ee (127.1.46.193:38305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb02765efbef43a3812b6e1705744666: Network error: Client connection negotiation failed: client connection to 127.1.46.193:38305: connect: Connection refused (error 111)
I20260927 10:57:05.343068 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 14288
I20260927 10:57:05.352380 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 13920
I20260927 10:57:05.352615 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:05.352275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:46499 (local address 127.0.0.1:49692)
0927 10:57:05.352450 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:05.352516 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.195:46499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 10:57:05.361425 1211 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:57:05.695606 1211 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:57:05.695708 1211 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:57:06.039753 1211 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:57:06.039 UTC [12987] LOG: received smart shutdown request
2026-09-27 10:57:06.044 UTC [12987] LOG: background worker "logical replication launcher" (PID 12996) exited with exit code 1
2026-09-27 10:57:06.044 UTC [12991] LOG: shutting down
2026-09-27 10:57:06.196 UTC [12987] LOG: database system is shut down
2026-09-27T10:57:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46483 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:06 dist-test-slave-cffh krb5kdc[14491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:06 dist-test-slave-cffh krb5kdc[14491](info): set up 2 sockets
Sep 27 10:57:06 dist-test-slave-cffh krb5kdc[14491](info): commencing operation
krb5kdc: starting...
W20260927 10:57:07.334928 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:57:07.445161 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43525
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:43525,127.1.46.253:32793,127.1.46.252:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:07.550191 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:07.550473 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:07.550550 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:07.554410 14504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:07.554486 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:07.554535 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:07.554608 14504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:07.554648 14504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:07.559906 14504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:43525,127.1.46.253:32793,127.1.46.252:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:07.560832 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:07.561762 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:07.569492 14512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:07.569499 14509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:07.569556 14504 server_base.cc:1034] running on GCE node
W20260927 10:57:07.569509 14510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:07.570313 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:07.571316 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:07.572489 14504 hybrid_clock.cc:648] HybridClock initialized: now 1790506627572477 us; error 27 us; skew 500 ppm
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:07.575181 14504 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:07.576030 14504 webserver.cc:460] Webserver started at http://127.1.46.254:42671/ using document root <none> and password file <none>
I20260927 10:57:07.576534 14504 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:07.576592 14504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:07.576807 14504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:07.578758 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "62aab1d97fc543ce8d6c5c7538478449"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "6d459a73379b610732949ec35c2eb741"
server_key_iv: "d6aa2463acf5b64bd310f4be1fe4f8c4"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.579291 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "62aab1d97fc543ce8d6c5c7538478449"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "6d459a73379b610732949ec35c2eb741"
server_key_iv: "d6aa2463acf5b64bd310f4be1fe4f8c4"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.582543 14504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:57:07.584807 14519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:07.585929 14504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:57:07.586071 14504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "62aab1d97fc543ce8d6c5c7538478449"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "6d459a73379b610732949ec35c2eb741"
server_key_iv: "d6aa2463acf5b64bd310f4be1fe4f8c4"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.586169 14504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:07.611774 14504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:07.614575 14504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:07.614787 14504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:07.621858 14504 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:43525
I20260927 10:57:07.621906 14571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:43525 every 8 connection(s)
I20260927 10:57:07.622812 14504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:07.625499 14572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:07.629272 14572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:07.630318 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 14504
I20260927 10:57:07.630460 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:57:07.630884 1211 external_mini_cluster.cc:1428] Setting key 476fb0591db14b2d18beb4e976049d6b
WARNING: no policy specified for kudu/127.1.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.253@KRBTEST.COM" created.
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:07.647938 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.631725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.1.46.254:43525)
0927 10:57:07.632188 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:07.632208 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:07.632540 (+ 332us) server_negotiation.cc:407] Connection header received
0927 10:57:07.633461 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:07.633470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:07.633701 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:07.633866 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:07.635073 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.635946 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.637440 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.637819 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.637986 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.642895 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:07.642900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:07.642908 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:07.642927 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:07.644852 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.645360 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.645369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.645374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.645510 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.645895 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.645902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.645903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.646401 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:07.646503 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:07.647011 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 10:57:07.647220 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260927 10:57:07.647940 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.631725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:43525 (local address 127.0.0.1:35954)
0927 10:57:07.632282 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.632327 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:07.633107 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:07.633933 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:07.633941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:07.634408 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:07.634838 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.634860 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.636081 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.636087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.637303 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.637318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.638012 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.638017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.638208 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.638872 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:07.638892 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:07.642749 (+ 3857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:07.644982 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.644991 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.644998 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.645237 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.645635 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.645638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.645641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.645789 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.646543 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:07.646551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:07.646751 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 10:57:07.646961 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 10:57:07.647355 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":84,"spinlock_wait_cycles":54016,"thread_start_us":150,"threads_started":1}
I20260927 10:57:07.650216 14572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:07.650942 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.650710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46280)
0927 10:57:07.650833 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.650896 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 10:57:07.651129 14521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
I20260927 10:57:07.651978 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.651800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46282)
0927 10:57:07.651894 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.651934 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:57:07.652115 14572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:32793: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab.
Entry for principal kudu/127.1.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab.
I20260927 10:57:07.657590 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:32793
--webserver_interface=127.1.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab
--principal=kudu/127.1.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:43525,127.1.46.253:32793,127.1.46.252:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:07.701344 14572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } attempt: 1
I20260927 10:57:07.702168 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.701823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46292)
0927 10:57:07.701991 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.702071 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 10:57:07.703011 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.702770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46304)
0927 10:57:07.702888 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.702944 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:57:07.703159 14572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:32793: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
W20260927 10:57:07.766440 14579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:07.766760 14579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:07.766810 14579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:07.770761 14579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:07.770812 14579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:07.770831 14579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:07.770848 14579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:07.770862 14579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:07.771322 14572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } attempt: 2
I20260927 10:57:07.772094 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.771775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46312)
0927 10:57:07.771940 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.772009 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 10:57:07.772743 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.772545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46326)
0927 10:57:07.772644 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.772689 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:57:07.772897 14572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:32793: Network error: Client connection negotiation failed: client connection to 127.1.46.253:32793: connect: Connection refused (error 111)
I20260927 10:57:07.776417 14579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:43525,127.1.46.253:32793,127.1.46.252:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab
--principal=kudu/127.1.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:07.777289 14579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:07.778177 14579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:07.785861 14584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:07.786011 14579 server_base.cc:1034] running on GCE node
W20260927 10:57:07.785919 14585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:07.785854 14587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:07.786669 14579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:07.787654 14579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:07.788836 14579 hybrid_clock.cc:648] HybridClock initialized: now 1790506627788821 us; error 27 us; skew 500 ppm
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:07.791684 14579 init.cc:377] Logged in from keytab as kudu/127.1.46.253@KRBTEST.COM (short username kudu)
I20260927 10:57:07.792451 14579 webserver.cc:460] Webserver started at http://127.1.46.253:37995/ using document root <none> and password file <none>
I20260927 10:57:07.792956 14579 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:07.793042 14579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:07.793267 14579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:07.795231 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data/instance:
uuid: "19b7c0c770b04fca9df849fdf6182b31"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "8e8d3073e1d1a67bf6531f67ff5bd3d4"
server_key_iv: "8284420ef1334742129e070f3892e6d3"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.795773 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/wal/instance:
uuid: "19b7c0c770b04fca9df849fdf6182b31"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "8e8d3073e1d1a67bf6531f67ff5bd3d4"
server_key_iv: "8284420ef1334742129e070f3892e6d3"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.799264 14579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:07.801548 14594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:07.802699 14579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:07.802819 14579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/wal
uuid: "19b7c0c770b04fca9df849fdf6182b31"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "8e8d3073e1d1a67bf6531f67ff5bd3d4"
server_key_iv: "8284420ef1334742129e070f3892e6d3"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.802918 14579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:07.811038 14579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:07.813827 14579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:07.814009 14579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:07.820942 14579 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.253:32793
I20260927 10:57:07.820963 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.253:32793 every 8 connection(s)
I20260927 10:57:07.821776 14579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
I20260927 10:57:07.823053 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 14579
I20260927 10:57:07.823172 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-1/wal/instance
I20260927 10:57:07.823498 1211 external_mini_cluster.cc:1428] Setting key a4a71a59cbfb8c51dc79354dd571f9fe
I20260927 10:57:07.824519 14647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:07.828428 14647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.252@KRBTEST.COM" created.
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:07.841856 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.830164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.1.46.254:43525)
0927 10:57:07.830324 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:07.830328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:07.831317 (+ 989us) server_negotiation.cc:407] Connection header received
0927 10:57:07.832172 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:07.832176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:07.832259 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:07.832360 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:07.833502 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.834034 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.835528 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.835760 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.835847 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.838399 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:07.838406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:07.838411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:07.838442 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:07.840043 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.840510 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.840516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.840520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.840614 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.840927 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.840932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.840934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.841143 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:07.841240 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:07.841650 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 10:57:07.841742 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:57:07.842540 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.830649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:43525 (local address 127.0.0.1:35958)
0927 10:57:07.831124 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.831162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:07.831977 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:07.832517 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:07.832523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:07.832823 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:07.833299 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.833336 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.834161 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.834166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.835399 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.835412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.835907 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.835910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.836079 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.836677 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:07.836692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:07.838227 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:07.840153 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.840158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.840167 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.840411 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.840702 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.840704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.840705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.840826 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.841237 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:07.841241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:07.841436 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 10:57:07.841577 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 10:57:07.841801 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":128,"threads_started":1}
I20260927 10:57:07.844043 14647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab.
Entry for principal kudu/127.1.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab.
I20260927 10:57:07.848721 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:34901
--webserver_interface=127.1.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab
--principal=kudu/127.1.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:40045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:43525,127.1.46.253:32793,127.1.46.252:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:57:07.854279 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.844536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46334)
0927 10:57:07.844644 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.844657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:07.844730 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:07.845341 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:07.845343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:07.845485 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:07.845563 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.845568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.846766 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.846772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.847379 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.847389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.848079 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.848086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.848221 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.848698 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:07.848709 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:07.850178 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:07.852643 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.852646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.852649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.852935 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.853255 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.853260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.853262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.853357 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.853781 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:07.853783 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:07.853866 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 10:57:07.853997 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 10:57:07.854113 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:57:07.854418 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.844633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.1.46.253:32793)
0927 10:57:07.844948 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:07.844964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:07.844982 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:57:07.845013 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:07.845016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:07.845144 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:07.845287 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:07.845670 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.846510 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.847500 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.847928 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.848055 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.850338 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:07.850347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:07.850359 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:07.850383 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:07.852526 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.853041 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.853047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.853049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.853158 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.853466 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.853470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.853472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.853682 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:07.853811 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:07.854087 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 10:57:07.854200 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":125,"threads_started":1}
I20260927 10:57:07.856251 14647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:07.857008 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.856666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52176)
0927 10:57:07.856827 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.856920 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 10:57:07.857199 14598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
I20260927 10:57:07.858047 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.857775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52192)
0927 10:57:07.857906 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.857971 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:57:07.858229 14647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:34901: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
I20260927 10:57:07.864095 14572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } attempt: 3
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:57:07.873930 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.864616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46338)
0927 10:57:07.864724 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.864742 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:07.864843 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:07.865087 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:07.865089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:07.865279 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:07.865761 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.865770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.867449 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.867452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.868017 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:07.868026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.868743 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.868747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:07.868839 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.869330 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:07.869343 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:07.870665 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:07.872546 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.872548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.872550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.872741 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.873092 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:07.873095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:07.873096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:07.873170 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:07.873555 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:07.873557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:07.873633 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 10:57:07.873712 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 10:57:07.873808 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 10:57:07.874046 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.864671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.1.46.253:32793)
0927 10:57:07.864807 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:07.864811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:07.864826 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:57:07.864935 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:07.864938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:07.864982 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:07.865066 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:07.865918 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.867327 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.868130 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:07.868539 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:07.868651 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:07.870816 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:07.870822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:07.870824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:07.870853 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:07.872433 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.872904 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.872908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.872910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.872992 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:07.873255 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:07.873258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:07.873259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:07.873462 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:07.873562 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:07.873785 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 10:57:07.873911 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:57:07.875049 14572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:07.875702 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.875500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52196)
0927 10:57:07.875609 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.875653 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:57:07.876417 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.876189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52210)
0927 10:57:07.876321 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.876375 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 10:57:07.876609 14572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:34901: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
I20260927 10:57:07.907483 14647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } attempt: 1
I20260927 10:57:07.907752 14572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } attempt: 1
I20260927 10:57:07.908356 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.908027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52222)
0927 10:57:07.908199 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.908263 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 10:57:07.908694 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.908249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52228)
0927 10:57:07.908397 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.908608 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:57:07.909047 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.908829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52242)
0927 10:57:07.908942 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.908990 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 10:57:07.909210 14647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:34901: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
I20260927 10:57:07.909452 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.909229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52256)
0927 10:57:07.909343 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.909393 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:57:07.909584 14572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:34901: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
W20260927 10:57:07.963435 14653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:07.963668 14653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:07.963706 14653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:07.967684 14653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:07.967737 14653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:07.967756 14653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:07.967772 14653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:07.967787 14653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:07.972888 14653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:40045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:43525,127.1.46.253:32793,127.1.46.252:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab
--principal=kudu/127.1.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:07.973875 14653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:07.974869 14653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:57:07.977370 14647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } attempt: 2
I20260927 10:57:07.978185 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.977873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52266)
0927 10:57:07.978030 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.978099 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 10:57:07.979031 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.978769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52278)
0927 10:57:07.978915 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.978969 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 10:57:07.979277 14647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:34901: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
W20260927 10:57:07.982954 14660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:07.982946 14662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:07.982946 14659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:07.983361 14653 server_base.cc:1034] running on GCE node
I20260927 10:57:07.983919 14653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:07.984737 14572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } attempt: 2
I20260927 10:57:07.985004 14653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:07.985435 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.985157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52290)
0927 10:57:07.985310 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.985369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:57:07.986065 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:07.985859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52302)
0927 10:57:07.985959 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:07.986007 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:57:07.986202 14572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:34901: Network error: Client connection negotiation failed: client connection to 127.1.46.252:34901: connect: Connection refused (error 111)
I20260927 10:57:07.986238 14653 hybrid_clock.cc:648] HybridClock initialized: now 1790506627986211 us; error 28 us; skew 500 ppm
Sep 27 10:57:07 dist-test-slave-cffh krb5kdc[14491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:07.989001 14653 init.cc:377] Logged in from keytab as kudu/127.1.46.252@KRBTEST.COM (short username kudu)
I20260927 10:57:07.989820 14653 webserver.cc:460] Webserver started at http://127.1.46.252:34787/ using document root <none> and password file <none>
I20260927 10:57:07.990373 14653 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:07.990458 14653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:07.990715 14653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:07.992534 14653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data/instance:
uuid: "3d4230d95968438ebf2280fe4035337f"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "214284b1003f819b91d3d1e4f9be0d56"
server_key_iv: "7d09d16cbe0e516e50ddbdf3541202c3"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.993053 14653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/wal/instance:
uuid: "3d4230d95968438ebf2280fe4035337f"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "214284b1003f819b91d3d1e4f9be0d56"
server_key_iv: "7d09d16cbe0e516e50ddbdf3541202c3"
server_key_version: "encryptionkey@0"
I20260927 10:57:07.996428 14653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:07.998703 14669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:07.999848 14653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:07.999980 14653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/wal
uuid: "3d4230d95968438ebf2280fe4035337f"
format_stamp: "Formatted at 2026-09-27 10:57:07 on dist-test-slave-cffh"
server_key: "214284b1003f819b91d3d1e4f9be0d56"
server_key_iv: "7d09d16cbe0e516e50ddbdf3541202c3"
server_key_version: "encryptionkey@0"
I20260927 10:57:08.000083 14653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:08.035065 14653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:08.038129 14653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:08.038321 14653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:08.045292 14653 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.252:34901
I20260927 10:57:08.045385 14721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.252:34901 every 8 connection(s)
I20260927 10:57:08.046159 14653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
I20260927 10:57:08.048776 14722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:08.052255 14722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:08.055416 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 14653
I20260927 10:57:08.055546 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790506472751632-1211-0/minicluster-data/master-2/wal/instance
I20260927 10:57:08.055897 1211 external_mini_cluster.cc:1428] Setting key 0b68ae9b2a15abb1bbf9fbced394277c
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:57:08.066344 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.053953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.1.46.254:43525)
0927 10:57:08.054097 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.054100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.054918 (+ 818us) server_negotiation.cc:407] Connection header received
0927 10:57:08.056105 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.056108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.056155 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.056254 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.057445 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.058047 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.059198 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.059421 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.059533 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.062236 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.062240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.062243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.062274 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.064111 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.064727 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.064730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.064731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.064803 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.065146 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.065149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.065151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.065368 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.065479 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.066057 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.066182 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:57:08.067257 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.054324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:43525 (local address 127.0.0.1:35964)
0927 10:57:08.054783 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.054823 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:08.055843 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:08.056403 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:08.056410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:08.056848 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:08.057294 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.057306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.058159 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.058164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.059077 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.059086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.059560 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.059564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.059744 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.060305 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:08.060322 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:08.062094 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:08.064214 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.064222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.064232 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.064628 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.064890 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.064895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.064899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.065052 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.065497 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:08.065502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:08.065794 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 10:57:08.065996 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 10:57:08.066267 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":99,"threads_started":1}
I20260927 10:57:08.067270 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.056585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.1.46.252:34901)
0927 10:57:08.056955 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.056966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.056990 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:57:08.057038 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.057044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.057201 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.057355 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.058140 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.059133 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.059978 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.060333 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.060444 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.063291 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.063296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.063304 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.063341 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.065039 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.065497 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.065502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.065504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.065582 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.065818 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.065821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.065822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.066064 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.066222 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.066636 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.066761 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
I20260927 10:57:08.068500 14722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:08.070446 14647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } attempt: 3
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:57:08.078019 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.069042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:32793 (local address 127.0.0.1:46340)
0927 10:57:08.069126 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.069140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:08.069205 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:08.069522 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:08.069526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:08.069686 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:08.069785 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.069794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.070794 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.070797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.071572 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.071581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.071992 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.071995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.072093 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.072553 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:08.072567 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:08.074003 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:08.076127 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.076131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.076134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.076369 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.077031 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.077035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.077037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.077129 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.077560 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:08.077563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:08.077659 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 10:57:08.077760 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:57:08.077874 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 10:57:08.078071 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.069170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.1.46.253:32793)
0927 10:57:08.069303 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.069308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.069322 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:57:08.069363 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.069366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.069415 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.069495 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.069927 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.070683 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.071682 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.071883 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.071991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.074135 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.074139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.074143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.074166 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.076018 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.076837 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.076841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.076843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.076922 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.077256 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.077260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.077263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.077461 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.077537 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.077838 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.077937 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:57:08.079231 14722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:57:08.079658 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.069283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.1.46.254:43525)
0927 10:57:08.069413 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.069417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.069436 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:57:08.069541 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.069544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.069589 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.069669 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.070339 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.071034 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.071966 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.072278 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.072403 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.075431 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.075435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.075439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.075462 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.077574 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.078354 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.078357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.078358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.078424 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.078738 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.078744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.078746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.078959 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.079220 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.079372 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.079492 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:57:08.081409 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.070842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52324)
0927 10:57:08.070960 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.070975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:08.071044 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:08.072224 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:08.072230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:08.072467 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:08.072573 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.072580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.073405 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.073410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.073986 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.073997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.074726 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.074730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.074817 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.075245 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:08.075255 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:08.076691 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:08.079464 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.079467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.079470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.079698 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.080035 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.080038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.080040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.080127 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.080860 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:08.080866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:08.080981 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 10:57:08.081079 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:57:08.081218 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:57:08.081895 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.071060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.1.46.252:34901)
0927 10:57:08.072034 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.072037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.072047 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:57:08.072074 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.072077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.072109 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.072186 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.072701 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.073292 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.074117 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.074525 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.074725 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.076823 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.076828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.076830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.076853 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.079345 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.079844 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.079847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.079848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.079929 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.080242 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.080245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.080247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.080738 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.081565 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.081634 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.081747 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":909}
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:57:08.085352 14572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } attempt: 3
I20260927 10:57:08.085469 14647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: Bootstrap starting.
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:57:08.088518 14647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:08.088845 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.079672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52330)
0927 10:57:08.080261 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.080275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:08.080358 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:08.080572 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:08.080575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:08.080739 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:08.080833 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.080840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.082207 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.082213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.083005 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.083014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.083643 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.083646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.083713 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.084112 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:08.084125 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:08.085416 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:08.087374 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.087376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.087378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.087581 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.087840 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.087846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.087851 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.087948 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.088408 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:08.088411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:08.088477 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 10:57:08.088558 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:57:08.088660 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526}
I20260927 10:57:08.089361 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.079886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.1.46.252:34901)
0927 10:57:08.080218 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.080228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.080339 (+ 111us) server_negotiation.cc:407] Connection header received
0927 10:57:08.080426 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.080428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.080460 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.080557 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.080952 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.082105 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.083125 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.083469 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.083559 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.085549 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.085552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.085554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.085574 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.087267 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.087675 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.087677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.087679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.087748 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.088070 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.088073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.088074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.088288 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.088377 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.089104 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.089206 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":257,"thread_start_us":98,"threads_started":1}
I20260927 10:57:08.089924 14647 log.cc:824] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:08.090523 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.081077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.1.46.253:32793)
0927 10:57:08.081185 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.081189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.081201 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:57:08.081234 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.081236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.081265 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.081354 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.082174 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.082791 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.083504 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.083730 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.083837 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.086945 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.086949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.086951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.086978 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.088634 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.089130 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.089133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.089135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.089217 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.089517 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.089520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.089521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.089714 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.089890 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.090215 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.090345 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:57:08.092175 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: Bootstrap starting.
I20260927 10:57:08.092278 14647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: No bootstrap required, opened a new log
I20260927 10:57:08.093469 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:08.094871 14722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:08.096069 14722 log.cc:824] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:57:08 dist-test-slave-cffh krb5kdc[14491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:57:08.096639 14647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.096874 14647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:08.096980 14647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b7c0c770b04fca9df849fdf6182b31, State: Initialized, Role: FOLLOWER
I20260927 10:57:08.097489 14647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.099048 14647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:08.101567 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.085699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:34901 (local address 127.0.0.1:52336)
0927 10:57:08.086168 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:08.086182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:08.086258 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:08.086936 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:08.086939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:08.087119 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:08.087530 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.087541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.088806 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.088809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.089664 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:08.089674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.094971 (+ 5297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.094975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:08.095062 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.095662 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:08.095678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:08.097176 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:08.099994 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.099997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.099999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.100243 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.100537 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:08.100540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:08.100542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:08.100623 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:08.101041 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:08.101047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:08.101153 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:57:08.101250 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:57:08.101377 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260927 10:57:08.102185 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.086174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.1.46.252:34901)
0927 10:57:08.086681 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.086702 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.086717 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:57:08.086751 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.086753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.086799 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.087653 (+ 854us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.088007 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.088689 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.089854 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.094822 (+ 4968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.095326 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.097306 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.097312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.097316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.097336 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.099876 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.100358 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.100360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.100362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.100444 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.100723 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.100726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.100728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.100924 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.101845 (+ 921us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.101927 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.102031 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"spinlock_wait_cycles":1408}
I20260927 10:57:08.103053 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: No bootstrap required, opened a new log
I20260927 10:57:08.106182 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.096027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.1.46.252:34901)
0927 10:57:08.096141 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.096148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.097048 (+ 900us) server_negotiation.cc:407] Connection header received
0927 10:57:08.097225 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.097230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.097276 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.097322 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.098296 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.099118 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.100253 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.100623 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.100714 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.102262 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.102267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.102271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.102299 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.104301 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.104759 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.104762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.104764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.104867 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.105127 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.105130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.105132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.105307 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.105429 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.105948 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.106048 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":1792}
I20260927 10:57:08.102854 14579 master_runner.cc:186] Error getting master registration for 127.1.46.254:43525: OK, instance_id { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" instance_seqno: 1790506627619884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:57:08.106693 14579 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:57:08.108654 14572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449: Bootstrap starting.
W20260927 10:57:08.110040 14748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:08.110095 14748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:08.111025 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.096715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.1.46.253:32793)
0927 10:57:08.096809 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.096813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.098492 (+ 1679us) server_negotiation.cc:407] Connection header received
0927 10:57:08.098672 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.098675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.098715 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.098759 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.103125 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.103670 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.105373 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.105692 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.105759 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.108065 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.108069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.108073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.108097 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.109480 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.109882 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.109886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.109889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.109961 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.110226 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.110230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.110232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.110394 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.110533 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.110749 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.110862 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:57:08.111207 14572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:08.112035 14572 log.cc:824] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:08.113343 14722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.113632 14722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:08.113721 14572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449: No bootstrap required, opened a new log
I20260927 10:57:08.113723 14722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4230d95968438ebf2280fe4035337f, State: Initialized, Role: FOLLOWER
I20260927 10:57:08.114230 14722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.115521 14722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:08.118011 14572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.118254 14572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:08.118335 14572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62aab1d97fc543ce8d6c5c7538478449, State: Initialized, Role: FOLLOWER
I20260927 10:57:08.118816 14572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.120083 14572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:08.120448 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:08.095572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.1.46.254:43525)
0927 10:57:08.095669 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:08.095673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:08.097040 (+ 1367us) server_negotiation.cc:407] Connection header received
0927 10:57:08.097306 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:08.097309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:08.097353 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:08.097404 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:08.098965 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.099658 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.114729 (+ 15071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:08.115029 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:08.115098 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:08.116688 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:08.116691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:08.116694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:08.116716 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:08.118283 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.119127 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.119131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.119135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.119216 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:08.119523 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:08.119526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:08.119527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:08.119712 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:08.119828 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:08.120161 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 10:57:08.120276 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:57:08.121381 14751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.121515 14751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:08.121747 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.121907 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:08.119436 14653 master_runner.cc:186] Error getting master registration for 127.1.46.254:43525: OK, instance_id { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" instance_seqno: 1790506627619884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:57:08.123456 14653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:57:08.126077 14762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:08.126188 14762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:08.127954 14504 master_runner.cc:263] No leader master found from master 62aab1d97fc543ce8d6c5c7538478449
I20260927 10:57:08.128029 14504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:57:08.129184 14774 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:57:08.129249 14774 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:57:08.129748 14763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.129894 14763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:08.155624 14763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:08.155781 14763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.156733 14763 leader_election.cc:288] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19b7c0c770b04fca9df849fdf6182b31 (127.1.46.253:32793), 3d4230d95968438ebf2280fe4035337f (127.1.46.252:34901)
I20260927 10:57:08.157171 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62aab1d97fc543ce8d6c5c7538478449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7c0c770b04fca9df849fdf6182b31" is_pre_election: true
I20260927 10:57:08.157460 14621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62aab1d97fc543ce8d6c5c7538478449 in term 0.
I20260927 10:57:08.157981 14521 leader_election.cc:302] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b7c0c770b04fca9df849fdf6182b31, 62aab1d97fc543ce8d6c5c7538478449; no voters:
I20260927 10:57:08.158262 14763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:08.158344 14763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:08.158402 14763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:08.159667 14763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.159956 14763 leader_election.cc:288] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [CANDIDATE]: Term 1 election: Requested vote from peers 19b7c0c770b04fca9df849fdf6182b31 (127.1.46.253:32793), 3d4230d95968438ebf2280fe4035337f (127.1.46.252:34901)
I20260927 10:57:08.160544 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62aab1d97fc543ce8d6c5c7538478449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7c0c770b04fca9df849fdf6182b31"
I20260927 10:57:08.160656 14621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:08.161857 14621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62aab1d97fc543ce8d6c5c7538478449 in term 1.
I20260927 10:57:08.162189 14521 leader_election.cc:302] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b7c0c770b04fca9df849fdf6182b31, 62aab1d97fc543ce8d6c5c7538478449; no voters:
I20260927 10:57:08.163061 14763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:08.163003 14695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62aab1d97fc543ce8d6c5c7538478449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4230d95968438ebf2280fe4035337f"
I20260927 10:57:08.163235 14695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:08.163326 14763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [term 1 LEADER]: Becoming Leader. State: Replica: 62aab1d97fc543ce8d6c5c7538478449, State: Running, Role: LEADER
I20260927 10:57:08.163645 14763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } }
I20260927 10:57:08.164480 14695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62aab1d97fc543ce8d6c5c7538478449 in term 1.
I20260927 10:57:08.163005 14696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62aab1d97fc543ce8d6c5c7538478449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4230d95968438ebf2280fe4035337f" is_pre_election: true
I20260927 10:57:08.164783 14696 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 62aab1d97fc543ce8d6c5c7538478449 in term 1. Re-sending same reply.
I20260927 10:57:08.165249 14763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62aab1d97fc543ce8d6c5c7538478449. Latest consensus state: current_term: 1 leader_uuid: "62aab1d97fc543ce8d6c5c7538478449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.165350 14763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:08.168388 14779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:08.170651 14779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:08.175868 14696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [term 1 FOLLOWER]: Refusing update from remote peer 62aab1d97fc543ce8d6c5c7538478449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:08.176247 14621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [term 1 FOLLOWER]: Refusing update from remote peer 62aab1d97fc543ce8d6c5c7538478449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:08.176656 14777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:08.176792 14763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:08.180490 14751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62aab1d97fc543ce8d6c5c7538478449. Latest consensus state: current_term: 1 leader_uuid: "62aab1d97fc543ce8d6c5c7538478449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.180586 14751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:08.182745 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62aab1d97fc543ce8d6c5c7538478449. Latest consensus state: current_term: 1 leader_uuid: "62aab1d97fc543ce8d6c5c7538478449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.182881 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:08.187577 14777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62aab1d97fc543ce8d6c5c7538478449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.187685 14777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:08.188866 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62aab1d97fc543ce8d6c5c7538478449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.188961 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:08.189396 14780 mvcc.cc:205] Tried to move back new op lower bound from 7333915149001199616 to 7333915148960849920. Current Snapshot: MvccSnapshot[applied={T|T < 7333915149001199616}]
I20260927 10:57:08.189779 14783 mvcc.cc:205] Tried to move back new op lower bound from 7333915149001199616 to 7333915148960849920. Current Snapshot: MvccSnapshot[applied={T|T < 7333915149001199616}]
I20260927 10:57:08.190639 14763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62aab1d97fc543ce8d6c5c7538478449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.190748 14763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:08.191514 14751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62aab1d97fc543ce8d6c5c7538478449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62aab1d97fc543ce8d6c5c7538478449" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43525 } } peers { permanent_uuid: "19b7c0c770b04fca9df849fdf6182b31" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 32793 } } peers { permanent_uuid: "3d4230d95968438ebf2280fe4035337f" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 34901 } } }
I20260927 10:57:08.191623 14751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:57:08.193636 14781 mvcc.cc:205] Tried to move back new op lower bound from 7333915149001199616 to 7333915148960849920. Current Snapshot: MvccSnapshot[applied={T|T < 7333915149001199616}]
I20260927 10:57:08.195660 14779 catalog_manager.cc:1347] Generated new cluster ID: 7c1e6a4ea2524770b8623346acc52971
I20260927 10:57:08.195716 14779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:08.222532 14779 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:08.223762 14779 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:08.242180 14779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62aab1d97fc543ce8d6c5c7538478449: Generated new TSK 0
I20260927 10:57:08.243072 14779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:09.111398 14748 catalog_manager.cc:1259] Loaded cluster ID: 7c1e6a4ea2524770b8623346acc52971
I20260927 10:57:09.111485 14748 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: loading cluster ID for follower catalog manager: success
I20260927 10:57:09.113276 14748 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: acquiring CA information for follower catalog manager: success
I20260927 10:57:09.114421 14748 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19b7c0c770b04fca9df849fdf6182b31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:57:09.127239 14762 catalog_manager.cc:1259] Loaded cluster ID: 7c1e6a4ea2524770b8623346acc52971
I20260927 10:57:09.127302 14762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: loading cluster ID for follower catalog manager: success
I20260927 10:57:09.128804 14762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: acquiring CA information for follower catalog manager: success
I20260927 10:57:09.129814 14762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d4230d95968438ebf2280fe4035337f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:57:09.304894 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 14504
I20260927 10:57:09.311477 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 14579
I20260927 10:57:09.317072 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 14653
2026-09-27T10:57:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3004 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:09 dist-test-slave-cffh krb5kdc[14796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:09 dist-test-slave-cffh krb5kdc[14796](info): set up 2 sockets
Sep 27 10:57:09 dist-test-slave-cffh krb5kdc[14796](info): commencing operation
krb5kdc: starting...
W20260927 10:57:10.363775 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:57:10 dist-test-slave-cffh krb5kdc[14796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:57:10.479427 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40099
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:10.588374 14809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:10.588611 14809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:10.588652 14809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:10.592581 14809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:10.592634 14809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:10.592654 14809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:10.592708 14809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:10.592741 14809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:10.597609 14809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:10.598515 14809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:10.599501 14809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:10.607672 14815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:10.607666 14817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:10.607664 14814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:10.607847 14809 server_base.cc:1034] running on GCE node
I20260927 10:57:10.608554 14809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:10.609575 14809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:10.610761 14809 hybrid_clock.cc:648] HybridClock initialized: now 1790506630610738 us; error 40 us; skew 500 ppm
Sep 27 10:57:10 dist-test-slave-cffh krb5kdc[14796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:10.618541 14809 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:10.619417 14809 webserver.cc:460] Webserver started at http://127.1.46.254:36671/ using document root <none> and password file <none>
I20260927 10:57:10.619931 14809 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:10.619987 14809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:10.620148 14809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:10.622078 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "867c83f5872740dfa5b4dee86b2b6e97"
format_stamp: "Formatted at 2026-09-27 10:57:10 on dist-test-slave-cffh"
server_key: "724d28232b7e626c2f97fb9cfffc52eb"
server_key_iv: "5e9cddf2dba02f6888a24e62d1f97595"
server_key_version: "encryptionkey@0"
I20260927 10:57:10.622659 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "867c83f5872740dfa5b4dee86b2b6e97"
format_stamp: "Formatted at 2026-09-27 10:57:10 on dist-test-slave-cffh"
server_key: "724d28232b7e626c2f97fb9cfffc52eb"
server_key_iv: "5e9cddf2dba02f6888a24e62d1f97595"
server_key_version: "encryptionkey@0"
I20260927 10:57:10.626103 14809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:57:10.628507 14824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:10.629732 14809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:10.629832 14809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "867c83f5872740dfa5b4dee86b2b6e97"
format_stamp: "Formatted at 2026-09-27 10:57:10 on dist-test-slave-cffh"
server_key: "724d28232b7e626c2f97fb9cfffc52eb"
server_key_iv: "5e9cddf2dba02f6888a24e62d1f97595"
server_key_version: "encryptionkey@0"
I20260927 10:57:10.629931 14809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:10.643282 14809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:10.645826 14809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:10.645975 14809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:10.652499 14809 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:40099
I20260927 10:57:10.652514 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:40099 every 8 connection(s)
I20260927 10:57:10.653416 14809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:10.654645 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 14809
I20260927 10:57:10.654788 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:57:10.655104 1211 external_mini_cluster.cc:1428] Setting key 586702090154484605bdd1b6d5d678c1
I20260927 10:57:10.656601 14877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:10.662765 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97: Bootstrap starting.
Sep 27 10:57:10 dist-test-slave-cffh krb5kdc[14796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:10.665172 14877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:10.665889 14877 log.cc:824] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:10.667749 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97: No bootstrap required, opened a new log
I20260927 10:57:10.669235 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:10.656297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.1.46.254:40099)
0927 10:57:10.656724 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:10.656749 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:10.656782 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:57:10.657386 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:10.657401 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:10.657691 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:10.657964 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:10.658837 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:10.659837 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:10.661033 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:10.661477 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:10.661590 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:10.664373 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:10.664378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:10.664389 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:10.664414 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:10.666152 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:10.666719 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:10.666726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:10.666732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:10.666850 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:10.667162 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:10.667166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:10.667168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:10.667538 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:10.667734 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:10.668093 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:57:10.668350 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":161,"threads_started":1}
I20260927 10:57:10.670740 14877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867c83f5872740dfa5b4dee86b2b6e97" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40099 } }
I20260927 10:57:10.670961 14877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:10.671049 14877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867c83f5872740dfa5b4dee86b2b6e97, State: Initialized, Role: FOLLOWER
I20260927 10:57:10.671450 14877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867c83f5872740dfa5b4dee86b2b6e97" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40099 } }
I20260927 10:57:10.671602 14877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:10.671686 14877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:10.671806 14877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:10.672006 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:57:10.672822 14877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867c83f5872740dfa5b4dee86b2b6e97" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40099 } }
I20260927 10:57:10.673192 14877 leader_election.cc:302] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 867c83f5872740dfa5b4dee86b2b6e97; no voters:
I20260927 10:57:10.673496 14877 leader_election.cc:288] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:10.673602 14882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:10.673815 14882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [term 1 LEADER]: Becoming Leader. State: Replica: 867c83f5872740dfa5b4dee86b2b6e97, State: Running, Role: LEADER
I20260927 10:57:10.674145 14882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867c83f5872740dfa5b4dee86b2b6e97" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40099 } }
I20260927 10:57:10.674741 14877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:10.675801 14884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 867c83f5872740dfa5b4dee86b2b6e97. Latest consensus state: current_term: 1 leader_uuid: "867c83f5872740dfa5b4dee86b2b6e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867c83f5872740dfa5b4dee86b2b6e97" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40099 } } }
I20260927 10:57:10.675931 14884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:10.676262 14883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "867c83f5872740dfa5b4dee86b2b6e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867c83f5872740dfa5b4dee86b2b6e97" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40099 } } }
I20260927 10:57:10.676421 14883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:10.676321 14891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:10.679534 14891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:10.685472 14891 catalog_manager.cc:1347] Generated new cluster ID: f6c490edd0bd46ccbfe897614fd1e2f4
I20260927 10:57:10.685601 14891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:10.705336 14891 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:10.706197 14891 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:10.712263 14891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 867c83f5872740dfa5b4dee86b2b6e97: Generated new TSK 0
I20260927 10:57:10.712891 14891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:57:10.727566 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 14809
2026-09-27T10:57:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1408 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:10 dist-test-slave-cffh krb5kdc[14904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:10 dist-test-slave-cffh krb5kdc[14904](info): set up 2 sockets
Sep 27 10:57:10 dist-test-slave-cffh krb5kdc[14904](info): commencing operation
krb5kdc: starting...
W20260927 10:57:11.738813 1211 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:57:11 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:57:11.850804 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45349
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:11.956002 14917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:11.956270 14917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:11.956350 14917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:11.960245 14917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:11.960326 14917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:11.960364 14917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:11.960425 14917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:11.960461 14917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:11.965339 14917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:11.966382 14917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:11.967473 14917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:11.975731 14925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:11.975740 14922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:11.975739 14923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:11.975982 14917 server_base.cc:1034] running on GCE node
I20260927 10:57:11.976544 14917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:11.977567 14917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:11.978740 14917 hybrid_clock.cc:648] HybridClock initialized: now 1790506631978715 us; error 46 us; skew 500 ppm
Sep 27 10:57:11 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:11.981247 14917 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:11.982081 14917 webserver.cc:460] Webserver started at http://127.1.46.254:43731/ using document root <none> and password file <none>
I20260927 10:57:11.982688 14917 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:11.982777 14917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:11.983021 14917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:11.984884 14917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-cffh"
server_key: "3575ed6625e5f64dcab37411eff85acf"
server_key_iv: "df00034c3edf119d63ec9c91861913af"
server_key_version: "encryptionkey@0"
I20260927 10:57:11.985423 14917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-cffh"
server_key: "3575ed6625e5f64dcab37411eff85acf"
server_key_iv: "df00034c3edf119d63ec9c91861913af"
server_key_version: "encryptionkey@0"
I20260927 10:57:11.988837 14917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:57:11.991117 14932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:11.992184 14917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:11.992318 14917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-cffh"
server_key: "3575ed6625e5f64dcab37411eff85acf"
server_key_iv: "df00034c3edf119d63ec9c91861913af"
server_key_version: "encryptionkey@0"
I20260927 10:57:11.992419 14917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:12.010646 14917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:12.013201 14917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:12.013382 14917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:12.020238 14917 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45349
I20260927 10:57:12.020524 14984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45349 every 8 connection(s)
I20260927 10:57:12.021167 14917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:12.023854 14985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:12.025910 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 14917
I20260927 10:57:12.026041 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:57:12.026350 1211 external_mini_cluster.cc:1428] Setting key 1f5fc74c0fcfdc67e0995e3bc5d270e5
I20260927 10:57:12.030462 14985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Bootstrap starting.
I20260927 10:57:12.032902 14985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:12.033793 14985 log.cc:824] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:57:12 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:12.035665 14985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: No bootstrap required, opened a new log
I20260927 10:57:12.039216 14985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:12.039489 14985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:12.039577 14985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc7fbd541ef4c34989e9d2f88ab7a01, State: Initialized, Role: FOLLOWER
I20260927 10:57:12.039992 14985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:12.040134 14985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:12.040228 14985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:12.040305 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.027642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.1.46.254:45349)
0927 10:57:12.028014 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.028039 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.028071 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:57:12.028683 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.028698 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.028994 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.029245 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.030122 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.031181 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.032337 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.032773 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.032881 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.035644 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.035648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.035661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.035684 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.037309 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.037753 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.037763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.037767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.037863 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.038143 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.038149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.038151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.038542 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.038811 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.039051 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.039316 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
I20260927 10:57:12.040356 14985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:12.041204 14985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:12.041570 14985 leader_election.cc:302] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fc7fbd541ef4c34989e9d2f88ab7a01; no voters:
I20260927 10:57:12.041863 14985 leader_election.cc:288] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:12.041954 14990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:12.042291 14990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 LEADER]: Becoming Leader. State: Replica: 8fc7fbd541ef4c34989e9d2f88ab7a01, State: Running, Role: LEADER
I20260927 10:57:12.042634 14990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:12.043080 14985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:12.043643 14990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fc7fbd541ef4c34989e9d2f88ab7a01. Latest consensus state: current_term: 1 leader_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } } }
I20260927 10:57:12.043745 14990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:12.044574 14991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } } }
I20260927 10:57:12.044835 14991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:12.045305 14997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:12.047986 14997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:12.052887 14997 catalog_manager.cc:1347] Generated new cluster ID: 0566fe982117408c984f4a3ce03cbe16
I20260927 10:57:12.052946 14997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:12.071182 14997 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:12.072044 14997 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:12.079742 14997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Generated new TSK 0
I20260927 10:57:12.080394 14997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:57:12.107892 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45349
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:12.216404 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:12.216666 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:12.216747 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:12.220757 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:12.220845 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:12.220961 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:57:12.225843 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:12.226794 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:12.227763 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:12.236929 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.236964 15011 server_base.cc:1034] running on GCE node
W20260927 10:57:12.236965 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:12.236948 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.237558 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:12.238157 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:12.239303 15011 hybrid_clock.cc:648] HybridClock initialized: now 1790506632239274 us; error 44 us; skew 500 ppm
Sep 27 10:57:12 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:12.241974 15011 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:57:12.242933 15011 webserver.cc:460] Webserver started at http://127.1.46.193:37133/ using document root <none> and password file <none>
I20260927 10:57:12.243451 15011 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:12.243552 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:12.243760 15011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:12.245649 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "33d326baf6ca4b049bd0ec1329f149ed"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "5021a08e9ebaec607331dc779337cd8d"
server_key_iv: "e0949ce613b4aab111e5a342115a3602"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.246209 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "33d326baf6ca4b049bd0ec1329f149ed"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "5021a08e9ebaec607331dc779337cd8d"
server_key_iv: "e0949ce613b4aab111e5a342115a3602"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.249581 15011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:57:12.251802 15026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.252986 15011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:12.253103 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "33d326baf6ca4b049bd0ec1329f149ed"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "5021a08e9ebaec607331dc779337cd8d"
server_key_iv: "e0949ce613b4aab111e5a342115a3602"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.253199 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:12.271011 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:12.273802 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:12.273988 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:12.274534 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:12.275312 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:12.275380 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.275449 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:12.275498 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.284448 15011 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:45671
I20260927 10:57:12.284535 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:45671 every 8 connection(s)
I20260927 10:57:12.285327 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:12.293599 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15011
I20260927 10:57:12.293754 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:12.294061 1211 external_mini_cluster.cc:1428] Setting key 7a0b8aa4b490c64a591bf65db91de7a7
Sep 27 10:57:12 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:12.299000 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.287176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:45821 (local address 127.1.46.254:45349)
0927 10:57:12.287307 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.287310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.288082 (+ 772us) server_negotiation.cc:407] Connection header received
0927 10:57:12.289340 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.289344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.289412 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.289507 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.290617 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.291157 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.292296 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.292550 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.292666 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.295357 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.295361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.295363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.295392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.297007 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.297515 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.297520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.297522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.297604 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.297955 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.297960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.297962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.298131 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.298235 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.298719 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.298834 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:57:12.299753 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.287464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:45821)
0927 10:57:12.287904 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.287955 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.289123 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.289666 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.289672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.290017 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:12.290456 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.290468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.291320 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.291322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.292174 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.292182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.292705 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.292709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.292893 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.293480 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:12.293495 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:12.295165 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:12.297112 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.297116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.297125 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.297345 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.297697 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.297702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.297704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.297835 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.298234 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:12.298240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:12.298465 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.298678 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.298971 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
I20260927 10:57:12.300882 15140 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45349
I20260927 10:57:12.301124 15140 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:12.301684 15140 heartbeater.cc:511] Master 127.1.46.254:45349 requested a full tablet report, sending...
I20260927 10:57:12.303313 14949 ts_manager.cc:194] Registered new tserver with Master: 33d326baf6ca4b049bd0ec1329f149ed (127.1.46.193:45671)
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:57:12.304666 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:45821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:57:12.320793 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45349
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:12.427915 15145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:12.428151 15145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:12.428200 15145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:12.432130 15145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:12.432175 15145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:12.432251 15145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:57:12.437129 15145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:12.438047 15145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:12.438994 15145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:12.448549 15150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:12.448511 15151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:12.448525 15153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.448833 15145 server_base.cc:1034] running on GCE node
I20260927 10:57:12.449297 15145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:12.449903 15145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:12.451103 15145 hybrid_clock.cc:648] HybridClock initialized: now 1790506632451089 us; error 31 us; skew 500 ppm
Sep 27 10:57:12 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:12.453802 15145 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:57:12.454689 15145 webserver.cc:460] Webserver started at http://127.1.46.194:35253/ using document root <none> and password file <none>
I20260927 10:57:12.455206 15145 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:12.455289 15145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:12.455545 15145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:12.457587 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "58d1bf633f614d2fb698473974f2a671"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "4e3a86b56665ef975ed40dbbdb17d69b"
server_key_iv: "af20cbfb4ab726dd4a523a9ff8860395"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.458118 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "58d1bf633f614d2fb698473974f2a671"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "4e3a86b56665ef975ed40dbbdb17d69b"
server_key_iv: "af20cbfb4ab726dd4a523a9ff8860395"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.461645 15145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:57:12.464207 15160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.465425 15145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:12.465562 15145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "58d1bf633f614d2fb698473974f2a671"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "4e3a86b56665ef975ed40dbbdb17d69b"
server_key_iv: "af20cbfb4ab726dd4a523a9ff8860395"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.465667 15145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:12.481793 15145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:12.484972 15145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:12.485167 15145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:12.485716 15145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:12.486505 15145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:12.486572 15145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.486699 15145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:12.486752 15145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.495296 15145 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:45929
I20260927 10:57:12.495325 15273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:45929 every 8 connection(s)
I20260927 10:57:12.496269 15145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:57:12 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:12.506006 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15145
I20260927 10:57:12.506136 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:12.506421 1211 external_mini_cluster.cc:1428] Setting key 6410ac9f4c4fc5bd74fe2791f13dfcb1
I20260927 10:57:12.509004 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.497939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:58787 (local address 127.1.46.254:45349)
0927 10:57:12.498092 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.498096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.498872 (+ 776us) server_negotiation.cc:407] Connection header received
0927 10:57:12.500050 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.500053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.500115 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.500197 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.501251 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.501777 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.502891 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.503119 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.503203 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.505359 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.505362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.505364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.505385 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.506994 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.507479 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.507483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.507485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.507563 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.507973 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.507976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.507977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.508150 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.508226 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.508789 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.508877 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:57:12.510015 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.498314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:58787)
0927 10:57:12.498738 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.498776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.499863 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.500348 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.500354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.500672 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:12.501127 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.501139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.501880 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.501882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.502789 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.502798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.503240 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.503243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.503366 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.503835 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:12.503849 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:12.505221 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:12.507099 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.507105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.507118 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.507386 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.507674 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.507677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.507679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.507859 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.508261 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:12.508267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:12.508551 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.508755 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.509081 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260927 10:57:12.511111 15274 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45349
I20260927 10:57:12.511350 15274 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:12.511857 15274 heartbeater.cc:511] Master 127.1.46.254:45349 requested a full tablet report, sending...
I20260927 10:57:12.512892 14949 ts_manager.cc:194] Registered new tserver with Master: 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
I20260927 10:57:12.513446 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:58787
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:57:12.533113 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45349
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:12.639843 15279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:12.640067 15279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:12.640106 15279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:12.644441 15279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:12.644493 15279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:12.644582 15279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:57:12.649822 15279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:12.650789 15279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:12.651719 15279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:12.660748 15287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.660912 15279 server_base.cc:1034] running on GCE node
W20260927 10:57:12.660778 15284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:12.661098 15285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:12.661501 15279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:12.662060 15279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:12.663236 15279 hybrid_clock.cc:648] HybridClock initialized: now 1790506632663204 us; error 43 us; skew 500 ppm
Sep 27 10:57:12 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:12.665787 15279 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:57:12.666639 15279 webserver.cc:460] Webserver started at http://127.1.46.195:38659/ using document root <none> and password file <none>
I20260927 10:57:12.667151 15279 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:12.667238 15279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:12.667454 15279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:12.669488 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "bf4154861fa94c7bbe592e4ca2382b8b"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "6970fccbe4276754f9f0ca94ab7a5dbc"
server_key_iv: "320a850fa8bded407296422bca748b74"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.670027 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "bf4154861fa94c7bbe592e4ca2382b8b"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "6970fccbe4276754f9f0ca94ab7a5dbc"
server_key_iv: "320a850fa8bded407296422bca748b74"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.673401 15279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:57:12.675695 15294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.676743 15279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:12.676867 15279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "bf4154861fa94c7bbe592e4ca2382b8b"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "6970fccbe4276754f9f0ca94ab7a5dbc"
server_key_iv: "320a850fa8bded407296422bca748b74"
server_key_version: "encryptionkey@0"
I20260927 10:57:12.676973 15279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:12.719128 15279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:12.722332 15279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:12.722529 15279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:12.723117 15279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:12.723875 15279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:12.723944 15279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.724013 15279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:12.724064 15279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:12.732872 15279 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:42877
I20260927 10:57:12.732903 15407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:42877 every 8 connection(s)
I20260927 10:57:12.733731 15279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:12.738832 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15279
I20260927 10:57:12.738927 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:12.739218 1211 external_mini_cluster.cc:1428] Setting key 435ad6e1ce0d4d7ed3dae0be81507796
Sep 27 10:57:12 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506632, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:12.747411 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.735680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:42783 (local address 127.1.46.254:45349)
0927 10:57:12.735809 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.735812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.736601 (+ 789us) server_negotiation.cc:407] Connection header received
0927 10:57:12.737808 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.737811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.737854 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.737933 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.739109 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.739750 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.741033 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.741284 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.741407 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.744080 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.744083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.744086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.744111 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.745466 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.745946 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.745948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.745950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.746033 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.746394 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.746398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.746399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.746620 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.746713 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.747144 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.747283 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:57:12.748085 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.736030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:42783)
0927 10:57:12.736451 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.736490 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.737619 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.738057 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.738063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.738390 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:12.738930 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.738952 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.739972 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.739977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.740891 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.740904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.741446 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.741451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.741629 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.742266 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:12.742283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:12.743910 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:12.745577 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.745581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.745604 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.745831 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.746131 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:12.746133 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:12.746134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:12.746312 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:12.746706 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:12.746710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:12.746903 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.747051 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.747278 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
I20260927 10:57:12.749215 15408 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45349
I20260927 10:57:12.749429 15408 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:12.749858 15408 heartbeater.cc:511] Master 127.1.46.254:45349 requested a full tablet report, sending...
I20260927 10:57:12.750891 14949 ts_manager.cc:194] Registered new tserver with Master: bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877)
I20260927 10:57:12.751572 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:42783
I20260927 10:57:12.752832 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:12.761324 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.754586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.1.46.254:45349)
0927 10:57:12.754707 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.754712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.754934 (+ 222us) server_negotiation.cc:407] Connection header received
0927 10:57:12.755010 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.755012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.755043 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.755108 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:12.755760 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.756260 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.757103 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.757310 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.757380 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.758815 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:12.758818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:12.758821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:12.758841 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:12.760019 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.760376 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.760378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.760380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.760444 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:12.760671 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:12.760673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:12.760674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:12.760854 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:12.760953 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.761140 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.761221 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:57:12.763931 14948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:12.766898 14948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:12.780049 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.774766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:42877 (local address 127.0.0.1:46652)
0927 10:57:12.775062 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.775112 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.775232 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.775934 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.775941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.775963 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.776111 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.776118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.777290 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.777295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.778748 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.778758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.779521 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.779528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.779649 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.779689 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.779797 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.779900 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260927 10:57:12.780064 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.774927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.1.46.195:42877)
0927 10:57:12.775276 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.775290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.775315 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:57:12.775387 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.775391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.775599 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.775748 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.776242 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.777168 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.778844 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.779415 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.779523 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.779658 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.779835 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.779921 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":164,"threads_started":1}
I20260927 10:57:12.781101 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.775484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:45929 (local address 127.0.0.1:49474)
0927 10:57:12.776144 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.776158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.776230 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.776779 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.776783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.776800 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.776894 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.776899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.778054 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.778058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.779300 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.779310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.780707 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.780709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.780813 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.780838 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.780913 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.780980 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":88,"threads_started":1}
I20260927 10:57:12.781198 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.775654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.1.46.194:45929)
0927 10:57:12.776300 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.776312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.776335 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:57:12.776421 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.776426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.776610 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.776742 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.777018 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.777891 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.779437 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.780367 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.780503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.780644 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.780965 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.781061 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":119,"threads_started":1}
I20260927 10:57:12.782367 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.775812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45671 (local address 127.0.0.1:35530)
0927 10:57:12.777405 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.777418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.777505 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.777905 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.777908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.777925 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.778020 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.778025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.779590 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.779592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.780714 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.780725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.781919 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.781923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.782022 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.782045 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.782140 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.782217 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"thread_start_us":79,"threads_started":1}
I20260927 10:57:12.782423 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.776197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.1.46.193:45671)
0927 10:57:12.776550 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.776565 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.777519 (+ 954us) server_negotiation.cc:407] Connection header received
0927 10:57:12.777597 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.777602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.777761 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.777908 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.778240 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.779437 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.780842 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.781799 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.781904 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.782041 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.782168 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.782279 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":79,"threads_started":1}
I20260927 10:57:12.783473 15342 tablet_service.cc:1467] Processing CreateTablet for tablet b09183c6b3e54110958947504865fe60 (DEFAULT_TABLE table=test-table [id=467b0d7fb1574ffd96eab68c3e0671bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:12.783864 15208 tablet_service.cc:1467] Processing CreateTablet for tablet b09183c6b3e54110958947504865fe60 (DEFAULT_TABLE table=test-table [id=467b0d7fb1574ffd96eab68c3e0671bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:12.784431 15342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09183c6b3e54110958947504865fe60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:12.784642 15208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09183c6b3e54110958947504865fe60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:12.789105 15074 tablet_service.cc:1467] Processing CreateTablet for tablet b09183c6b3e54110958947504865fe60 (DEFAULT_TABLE table=test-table [id=467b0d7fb1574ffd96eab68c3e0671bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:12.790131 15074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09183c6b3e54110958947504865fe60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:12.790518 15425 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Bootstrap starting.
I20260927 10:57:12.792851 15425 tablet_bootstrap.cc:654] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:12.793506 15425 log.cc:824] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:12.794219 15426 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Bootstrap starting.
I20260927 10:57:12.795516 15425 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: No bootstrap required, opened a new log
I20260927 10:57:12.795719 15425 ts_tablet_manager.cc:1397] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:57:12.795941 15428 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap starting.
I20260927 10:57:12.796692 15426 tablet_bootstrap.cc:654] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:12.797451 15426 log.cc:824] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:12.798028 15428 tablet_bootstrap.cc:654] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:12.798820 15425 raft_consensus.cc:348] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.799057 15425 raft_consensus.cc:374] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:12.799127 15425 raft_consensus.cc:729] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d1bf633f614d2fb698473974f2a671, State: Initialized, Role: FOLLOWER
I20260927 10:57:12.799135 15428 log.cc:824] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:12.799374 15426 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: No bootstrap required, opened a new log
I20260927 10:57:12.799557 15426 ts_tablet_manager.cc:1397] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:57:12.799592 15425 consensus_queue.cc:260] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.800258 15274 heartbeater.cc:503] Master 127.1.46.254:45349 was elected leader, sending a full tablet report...
I20260927 10:57:12.800776 15425 ts_tablet_manager.cc:1428] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:57:12.801761 15428 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: No bootstrap required, opened a new log
I20260927 10:57:12.801929 15428 ts_tablet_manager.cc:1397] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:12.803300 15426 raft_consensus.cc:348] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.803550 15426 raft_consensus.cc:374] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:12.803607 15426 raft_consensus.cc:729] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4154861fa94c7bbe592e4ca2382b8b, State: Initialized, Role: FOLLOWER
I20260927 10:57:12.804013 15426 consensus_queue.cc:260] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.804906 15426 ts_tablet_manager.cc:1428] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:12.805219 15408 heartbeater.cc:503] Master 127.1.46.254:45349 was elected leader, sending a full tablet report...
I20260927 10:57:12.805465 15428 raft_consensus.cc:348] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.805658 15428 raft_consensus.cc:374] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:12.805720 15428 raft_consensus.cc:729] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d326baf6ca4b049bd0ec1329f149ed, State: Initialized, Role: FOLLOWER
I20260927 10:57:12.806059 15428 consensus_queue.cc:260] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.806653 15140 heartbeater.cc:503] Master 127.1.46.254:45349 was elected leader, sending a full tablet report...
I20260927 10:57:12.807245 15428 ts_tablet_manager.cc:1428] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:12.855568 15433 raft_consensus.cc:482] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:12.855803 15433 raft_consensus.cc:504] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.857128 15433 leader_election.cc:288] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
I20260927 10:57:12.861200 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.857342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:42877 (local address 127.1.46.193:57683)
0927 10:57:12.857625 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.857641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.857776 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.858236 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.858238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.858280 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.858426 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.858436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.859238 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.859241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.860096 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.860105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.860787 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.860792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.860876 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.860905 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.860974 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.861034 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":110,"threads_started":1}
I20260927 10:57:12.861220 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.857426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:57683 (local address 127.1.46.195:42877)
0927 10:57:12.857570 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.857573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.857778 (+ 205us) server_negotiation.cc:407] Connection header received
0927 10:57:12.858079 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.858082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.858151 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.858227 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.858556 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.859132 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.860223 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.860656 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.860743 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.860816 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.861012 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.861094 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:57:12.861286 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.857948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:36491 (local address 127.1.46.194:45929)
0927 10:57:12.858103 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.858106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.858154 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:57:12.858311 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.858314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.858361 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.858443 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:12.858763 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.859258 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.860300 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.860693 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.860827 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.860874 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.861102 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.861163 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 10:57:12.861413 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.857761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:45929 (local address 127.1.46.193:36491)
0927 10:57:12.858082 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.858096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:12.858199 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:12.858462 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:12.858464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:12.858478 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:12.858655 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.858662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.859366 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.859368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.860180 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:12.860193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.860835 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.860841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:12.860952 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.861000 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 10:57:12.861072 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:57:12.861131 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260927 10:57:12.862179 15228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09183c6b3e54110958947504865fe60" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d1bf633f614d2fb698473974f2a671" is_pre_election: true
I20260927 10:57:12.862185 15362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09183c6b3e54110958947504865fe60" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" is_pre_election: true
I20260927 10:57:12.862473 15228 raft_consensus.cc:2454] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 0.
I20260927 10:57:12.862466 15362 raft_consensus.cc:2454] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 0.
I20260927 10:57:12.862926 15027 leader_election.cc:302] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed, 58d1bf633f614d2fb698473974f2a671; no voters:
I20260927 10:57:12.863162 15433 raft_consensus.cc:2781] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:12.863232 15433 raft_consensus.cc:482] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:12.863260 15433 raft_consensus.cc:3037] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:12.864215 15433 raft_consensus.cc:504] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.864531 15433 leader_election.cc:288] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 election: Requested vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
I20260927 10:57:12.864840 15362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09183c6b3e54110958947504865fe60" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4154861fa94c7bbe592e4ca2382b8b"
I20260927 10:57:12.864952 15228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09183c6b3e54110958947504865fe60" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d1bf633f614d2fb698473974f2a671"
I20260927 10:57:12.865005 15362 raft_consensus.cc:3037] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:12.865113 15228 raft_consensus.cc:3037] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:12.866026 15362 raft_consensus.cc:2454] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 1.
I20260927 10:57:12.866098 15228 raft_consensus.cc:2454] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 1.
I20260927 10:57:12.866291 15027 leader_election.cc:302] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed, bf4154861fa94c7bbe592e4ca2382b8b; no voters:
I20260927 10:57:12.866490 15433 raft_consensus.cc:2781] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:12.866715 15433 raft_consensus.cc:686] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 1 LEADER]: Becoming Leader. State: Replica: 33d326baf6ca4b049bd0ec1329f149ed, State: Running, Role: LEADER
I20260927 10:57:12.867053 15433 consensus_queue.cc:237] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:12.870330 14949 catalog_manager.cc:5495] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d326baf6ca4b049bd0ec1329f149ed (127.1.46.193). New cstate: current_term: 1 leader_uuid: "33d326baf6ca4b049bd0ec1329f149ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260927 10:57:12.904397 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.901218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.1.46.193:45671)
0927 10:57:12.901353 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:12.901357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:12.901368 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:57:12.901432 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:12.901434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:12.901508 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:12.901594 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:12.901807 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.902326 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.903220 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:12.903423 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:12.903495 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:12.903728 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:12.904117 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:12.904202 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:12.904254 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 10:57:12.904297 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:57:12.909890 15228 raft_consensus.cc:1261] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Refusing update from remote peer 33d326baf6ca4b049bd0ec1329f149ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:12.909890 15362 raft_consensus.cc:1261] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Refusing update from remote peer 33d326baf6ca4b049bd0ec1329f149ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:12.910609 15433 consensus_queue.cc:1035] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:12.910784 15436 consensus_queue.cc:1035] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:12.917786 15440 mvcc.cc:205] Tried to move back new op lower bound from 7333915168392093696 to 7333915168226082816. Current Snapshot: MvccSnapshot[applied={T|T < 7333915168392093696}]
I20260927 10:57:12.926944 14949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:12.927093 14949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:12.929440 14949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Sending DeleteTablet for 3 replicas of tablet b09183c6b3e54110958947504865fe60
I20260927 10:57:12.930122 15342 tablet_service.cc:1514] Processing DeleteTablet for tablet b09183c6b3e54110958947504865fe60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:12 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260927 10:57:12.930331 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 15011
I20260927 10:57:12.930419 15074 tablet_service.cc:1514] Processing DeleteTablet for tablet b09183c6b3e54110958947504865fe60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:12 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:35530
I20260927 10:57:12.930611 15208 tablet_service.cc:1514] Processing DeleteTablet for tablet b09183c6b3e54110958947504865fe60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:12 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:49474
I20260927 10:57:12.931136 15449 tablet_replica.cc:330] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: stopping tablet replica
I20260927 10:57:12.931402 15449 raft_consensus.cc:2229] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:12.931658 15448 tablet_replica.cc:330] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: stopping tablet replica
I20260927 10:57:12.931708 15449 raft_consensus.cc:2258] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:12.931953 15448 raft_consensus.cc:2229] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:12.932255 15448 raft_consensus.cc:2258] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:12.933029 15449 ts_tablet_manager.cc:1905] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:12.933424 15448 ts_tablet_manager.cc:1905] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:12.935490 15448 ts_tablet_manager.cc:1918] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:12.935595 15448 log.cc:1196] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/b09183c6b3e54110958947504865fe60
I20260927 10:57:12.935875 15448 ts_tablet_manager.cc:1939] T b09183c6b3e54110958947504865fe60 P bf4154861fa94c7bbe592e4ca2382b8b: Deleting consensus metadata
I20260927 10:57:12.936400 15449 ts_tablet_manager.cc:1918] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:12.936502 15449 log.cc:1196] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/b09183c6b3e54110958947504865fe60
I20260927 10:57:12.936883 14933 catalog_manager.cc:4841] TS bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877): tablet b09183c6b3e54110958947504865fe60 (table test-table [id=467b0d7fb1574ffd96eab68c3e0671bb]) successfully deleted
I20260927 10:57:12.936955 15449 ts_tablet_manager.cc:1939] T b09183c6b3e54110958947504865fe60 P 58d1bf633f614d2fb698473974f2a671: Deleting consensus metadata
I20260927 10:57:12.938035 14933 catalog_manager.cc:4841] TS 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929): tablet b09183c6b3e54110958947504865fe60 (table test-table [id=467b0d7fb1574ffd96eab68c3e0671bb]) successfully deleted
W20260927 10:57:12.938392 14933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:45671 (error 108)
I20260927 10:57:12.938716 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 15145
I20260927 10:57:12.939733 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:12.939410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45671 (local address 127.0.0.1:35550)
0927 10:57:12.939561 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:12.939638 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:57:12.939893 14933 catalog_manager.cc:4584] TS 33d326baf6ca4b049bd0ec1329f149ed (127.1.46.193:45671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b09183c6b3e54110958947504865fe60: Network error: Client connection negotiation failed: client connection to 127.1.46.193:45671: connect: Connection refused (error 111)
I20260927 10:57:12.945567 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 15279
I20260927 10:57:12.950996 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 14917
I20260927 10:57:12.956993 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45349
--webserver_interface=127.1.46.254
--webserver_port=43731
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:13.061509 15451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:13.061760 15451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:13.061830 15451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:13.065663 15451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:13.065738 15451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:13.065775 15451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:13.065831 15451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:13.065871 15451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:13.071007 15451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:13.071964 15451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.072872 15451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.084499 15458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.084542 15460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.084568 15457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.084703 15451 server_base.cc:1034] running on GCE node
I20260927 10:57:13.085332 15451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.086391 15451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.087585 15451 hybrid_clock.cc:648] HybridClock initialized: now 1790506633087576 us; error 46 us; skew 500 ppm
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:13.090308 15451 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:13.091216 15451 webserver.cc:460] Webserver started at http://127.1.46.254:43731/ using document root <none> and password file <none>
I20260927 10:57:13.091754 15451 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.091821 15451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.095534 15451 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:13.097579 15467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.099028 15451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:13.099125 15451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-cffh"
server_key: "3575ed6625e5f64dcab37411eff85acf"
server_key_iv: "df00034c3edf119d63ec9c91861913af"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.099704 15451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.114284 15451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.117602 15451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.117823 15451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.125131 15451 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45349
I20260927 10:57:13.125403 15519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45349 every 8 connection(s)
I20260927 10:57:13.126212 15451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:13.130151 15520 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:57:13.131699 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Bootstrap starting.
I20260927 10:57:13.132974 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15451
I20260927 10:57:13.133448 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:45671
--local_ip_for_outbound_sockets=127.1.46.193
--tserver_master_addrs=127.1.46.254:45349
--webserver_port=37133
--webserver_interface=127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:13.148209 15520 log.cc:824] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.156361 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:57:13.156690 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Bootstrap complete.
I20260927 10:57:13.160714 15520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:13.161116 15520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc7fbd541ef4c34989e9d2f88ab7a01, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.161557 15520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:13.161691 15520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:13.161777 15520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:13.161890 15520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:57:13.163710 15520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:13.164016 15520 leader_election.cc:302] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fc7fbd541ef4c34989e9d2f88ab7a01; no voters:
I20260927 10:57:13.164354 15520 leader_election.cc:288] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:57:13.164429 15524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:57:13.164626 15524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 LEADER]: Becoming Leader. State: Replica: 8fc7fbd541ef4c34989e9d2f88ab7a01, State: Running, Role: LEADER
I20260927 10:57:13.164898 15524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:13.165529 15520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:13.165769 15525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } } }
I20260927 10:57:13.165904 15525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:13.166986 15528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:13.167302 15526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fc7fbd541ef4c34989e9d2f88ab7a01. Latest consensus state: current_term: 2 leader_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } } }
I20260927 10:57:13.167500 15526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:13.171711 15528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:13.172328 15528 catalog_manager.cc:1259] Loaded cluster ID: 0566fe982117408c984f4a3ce03cbe16
I20260927 10:57:13.172386 15528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:13.173744 15528 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 10:57:13.178599 15528 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 10:57:13.183709 15528 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:13.184326 15528 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Loaded TSK: 0
I20260927 10:57:13.185436 15528 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:57:13.247463 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:13.247761 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:13.247843 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:13.251919 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:13.252009 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:13.252110 15522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:57:13.257539 15522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:45671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=37133
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:13.258549 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.259603 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.269131 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.269131 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.269217 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.269620 15522 server_base.cc:1034] running on GCE node
I20260927 10:57:13.270046 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.270776 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.271970 15522 hybrid_clock.cc:648] HybridClock initialized: now 1790506633271953 us; error 33 us; skew 500 ppm
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:13.274824 15522 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:57:13.275789 15522 webserver.cc:460] Webserver started at http://127.1.46.193:37133/ using document root <none> and password file <none>
I20260927 10:57:13.276336 15522 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.276412 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.280387 15522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:57:13.282612 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.283955 15522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:57:13.284085 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "33d326baf6ca4b049bd0ec1329f149ed"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "5021a08e9ebaec607331dc779337cd8d"
server_key_iv: "e0949ce613b4aab111e5a342115a3602"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.284643 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.296684 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.300074 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.300274 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.300895 15522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:13.302093 15564 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:57:13.306366 15522 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:57:13.306433 15522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 10:57:13.306483 15522 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:57:13.308166 15522 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:57:13.308231 15522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:57:13.308928 15564 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap starting.
I20260927 10:57:13.319993 15522 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:45671
I20260927 10:57:13.320111 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:45671 every 8 connection(s)
I20260927 10:57:13.321326 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:13.329965 15564 log.cc:824] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.330042 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15522
I20260927 10:57:13.330638 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:45929
--local_ip_for_outbound_sockets=127.1.46.194
--tserver_master_addrs=127.1.46.254:45349
--webserver_port=35253
--webserver_interface=127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:13.338025 15564 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:57:13.338461 15564 tablet_bootstrap.cc:492] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap complete.
I20260927 10:57:13.339637 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.323464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:55891 (local address 127.1.46.254:45349)
0927 10:57:13.323989 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.324009 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.324056 (+ 47us) server_negotiation.cc:407] Connection header received
0927 10:57:13.325222 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.325238 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.325634 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.326076 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.327063 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.328711 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.329868 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.330464 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.330648 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.333777 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.333789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.333803 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:13.333835 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.335912 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.336568 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.336574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.336579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.336720 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.337156 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.337159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.337161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.337584 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.337724 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.338449 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.338809 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":136,"threads_started":1}
I20260927 10:57:13.339680 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.323383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:55891)
0927 10:57:13.323852 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.323904 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.324758 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.326075 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.326082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.326407 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:13.326894 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.326906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.328856 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.328858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.329746 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.329755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.330641 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.330645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.330875 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.331637 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:13.331665 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:13.333574 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:13.336042 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.336051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.336066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.336441 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.336832 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.336838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.336843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.337044 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.337725 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:13.337734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:13.338064 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.338306 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.338678 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260927 10:57:13.339825 15564 ts_tablet_manager.cc:1397] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260927 10:57:13.342437 15672 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45349
I20260927 10:57:13.342713 15672 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:13.342278 15564 raft_consensus.cc:348] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.342829 15564 raft_consensus.cc:729] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d326baf6ca4b049bd0ec1329f149ed, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.343271 15672 heartbeater.cc:511] Master 127.1.46.254:45349 requested a full tablet report, sending...
I20260927 10:57:13.343323 15564 consensus_queue.cc:260] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.344278 15564 ts_tablet_manager.cc:1428] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 10:57:13.345659 15484 ts_manager.cc:194] Registered new tserver with Master: 33d326baf6ca4b049bd0ec1329f149ed (127.1.46.193:45671)
I20260927 10:57:13.348735 15484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:55891
I20260927 10:57:13.350943 15672 heartbeater.cc:503] Master 127.1.46.254:45349 was elected leader, sending a full tablet report...
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:57:13.357641 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.348219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45671 (local address 127.0.0.1:35560)
0927 10:57:13.348571 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.348611 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.348728 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.349467 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.349472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.349786 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:13.349869 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.349875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.350767 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.350772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.351524 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.351532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.352253 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.352256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.352360 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.352869 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:13.352880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:13.354269 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:13.356180 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.356187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.356191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.356390 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.356667 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.356669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.356670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.356739 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.357198 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:13.357203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:13.357289 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.357395 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.357528 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":105,"threads_started":1}
I20260927 10:57:13.357775 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.348286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.1.46.193:45671)
0927 10:57:13.348973 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.348984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.349005 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:57:13.349080 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.349084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.349249 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.349377 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.349986 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.350671 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.351631 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.352141 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.352239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.354393 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.354402 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.354413 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:13.354436 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.356084 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.356492 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.356497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.356499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.356582 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.356821 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.356824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.356826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.357085 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.357202 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.357493 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.357638 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":85,"threads_started":1}
I20260927 10:57:13.358649 15606 tablet_service.cc:1514] Processing DeleteTablet for tablet b09183c6b3e54110958947504865fe60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:12 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:35560
I20260927 10:57:13.359352 15681 tablet_replica.cc:330] stopping tablet replica
I20260927 10:57:13.359607 15681 raft_consensus.cc:2229] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:13.359822 15681 raft_consensus.cc:2258] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:13.360689 15681 ts_tablet_manager.cc:1905] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:13.363965 15681 ts_tablet_manager.cc:1918] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:13.364037 15681 log.cc:1196] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/b09183c6b3e54110958947504865fe60
I20260927 10:57:13.364322 15681 ts_tablet_manager.cc:1939] T b09183c6b3e54110958947504865fe60 P 33d326baf6ca4b049bd0ec1329f149ed: Deleting consensus metadata
I20260927 10:57:13.365922 15468 catalog_manager.cc:4841] TS 33d326baf6ca4b049bd0ec1329f149ed (127.1.46.193:45671): tablet b09183c6b3e54110958947504865fe60 (table test-table [id=467b0d7fb1574ffd96eab68c3e0671bb]) successfully deleted
W20260927 10:57:13.443898 15677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:13.444144 15677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:13.444216 15677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:13.448246 15677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:13.448318 15677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:13.448406 15677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:57:13.453538 15677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:45929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=35253
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:13.454421 15677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.455355 15677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.464156 15689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.464177 15686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.464201 15687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.464365 15677 server_base.cc:1034] running on GCE node
I20260927 10:57:13.464797 15677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.465500 15677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.466632 15677 hybrid_clock.cc:648] HybridClock initialized: now 1790506633466620 us; error 23 us; skew 500 ppm
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:13.469441 15677 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:57:13.470310 15677 webserver.cc:460] Webserver started at http://127.1.46.194:35253/ using document root <none> and password file <none>
I20260927 10:57:13.470884 15677 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.470955 15677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.474383 15677 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:13.476293 15696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.477468 15677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:13.477597 15677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "58d1bf633f614d2fb698473974f2a671"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "4e3a86b56665ef975ed40dbbdb17d69b"
server_key_iv: "af20cbfb4ab726dd4a523a9ff8860395"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.478092 15677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.496815 15677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.499876 15677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.500065 15677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.500581 15677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:13.501317 15677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:13.501384 15677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.501467 15677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:13.501518 15677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.510370 15677 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:45929
I20260927 10:57:13.510392 15809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:45929 every 8 connection(s)
I20260927 10:57:13.511334 15677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:13.518215 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15677
I20260927 10:57:13.518724 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:42877
--local_ip_for_outbound_sockets=127.1.46.195
--tserver_master_addrs=127.1.46.254:45349
--webserver_port=38659
--webserver_interface=127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:13.525586 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.513165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:55489 (local address 127.1.46.254:45349)
0927 10:57:13.513499 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.513503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.514107 (+ 604us) server_negotiation.cc:407] Connection header received
0927 10:57:13.515480 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.515483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.515546 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.515631 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.516758 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.517299 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.518698 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.519028 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.519119 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.522022 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.522025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.522027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:13.522050 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.523569 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.524105 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.524110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.524112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.524193 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.524543 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.524547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.524549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.524714 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.524815 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.525291 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.525414 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260927 10:57:13.526321 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.513490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:55489)
0927 10:57:13.513956 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.513997 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.515259 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.515798 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.515805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.516164 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:13.516610 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.516623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.517430 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.517437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.518555 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.518566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.519184 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.519191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.519429 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.520017 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:13.520042 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:13.521848 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:13.523729 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.523736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.523749 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.523994 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.524287 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.524292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.524295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.524445 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.524824 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:13.524832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:13.525052 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.525250 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.525574 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260927 10:57:13.527475 15810 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45349
I20260927 10:57:13.527719 15810 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:13.528151 15810 heartbeater.cc:511] Master 127.1.46.254:45349 requested a full tablet report, sending...
I20260927 10:57:13.529165 15484 ts_manager.cc:194] Registered new tserver with Master: 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
I20260927 10:57:13.529699 15484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:55489
W20260927 10:57:13.626024 15813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:13.626283 15813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:13.626360 15813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:13.630371 15813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:13.630455 15813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:13.630553 15813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:57:13.635622 15813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:42877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=38659
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:13.636575 15813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:13.637519 15813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:13.646616 15819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:13.646713 15818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.646760 15813 server_base.cc:1034] running on GCE node
W20260927 10:57:13.646685 15821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:13.647352 15813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:13.647864 15813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:13.649087 15813 hybrid_clock.cc:648] HybridClock initialized: now 1790506633649026 us; error 45 us; skew 500 ppm
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:13.651952 15813 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:57:13.652745 15813 webserver.cc:460] Webserver started at http://127.1.46.195:38659/ using document root <none> and password file <none>
I20260927 10:57:13.653359 15813 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:13.653491 15813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:13.657173 15813 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:13.659153 15828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.660384 15813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:13.660565 15813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "bf4154861fa94c7bbe592e4ca2382b8b"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "6970fccbe4276754f9f0ca94ab7a5dbc"
server_key_iv: "320a850fa8bded407296422bca748b74"
server_key_version: "encryptionkey@0"
I20260927 10:57:13.661109 15813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:13.685139 15813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:13.688012 15813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:13.688232 15813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:13.688804 15813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:13.689599 15813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:13.689673 15813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.689746 15813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:13.689805 15813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:13.699362 15813 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:42877
I20260927 10:57:13.699383 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:42877 every 8 connection(s)
I20260927 10:57:13.700354 15813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:13.705199 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15813
Sep 27 10:57:13 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:13.714831 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.702053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:39763 (local address 127.1.46.254:45349)
0927 10:57:13.702176 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.702179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.703070 (+ 891us) server_negotiation.cc:407] Connection header received
0927 10:57:13.704330 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.704334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.704389 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.704499 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.705600 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.706308 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.707471 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.707788 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.707907 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.710806 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.710809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.710811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:13.710834 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.712481 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.713108 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.713111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.713112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.713203 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.713630 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.713633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.713634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.713814 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.713962 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.714524 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.714676 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:57:13.715485 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.702454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:39763)
0927 10:57:13.702916 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.702954 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.704139 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.704665 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.704671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.704988 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:13.705464 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.705478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.706412 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.706417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.707350 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.707358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.707938 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.707941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.708109 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.708914 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:13.708932 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:13.710635 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:13.712593 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.712598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.712610 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.712938 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.713329 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:13.713334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:13.713337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:13.713476 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:13.713925 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:13.713933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:13.714223 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.714396 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.714685 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":177,"threads_started":1}
I20260927 10:57:13.716240 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.705878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.1.46.254:45349)
0927 10:57:13.706118 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.706121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.706289 (+ 168us) server_negotiation.cc:407] Connection header received
0927 10:57:13.707502 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.707505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.707549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.707628 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:13.708676 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.709411 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.710621 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.710907 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.710979 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.712606 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:13.712610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:13.712612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:13.712633 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:13.714179 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.714673 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.714677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.714679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.714764 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:13.715127 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:13.715132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:13.715135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:13.715306 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:13.715534 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.715862 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.715988 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260927 10:57:13.716588 15942 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45349
I20260927 10:57:13.716841 15942 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:13.717293 15942 heartbeater.cc:511] Master 127.1.46.254:45349 requested a full tablet report, sending...
I20260927 10:57:13.718283 15484 ts_manager.cc:194] Registered new tserver with Master: bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877)
I20260927 10:57:13.718889 15484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:39763
I20260927 10:57:13.719213 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:13.723358 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.719883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.1.46.254:45349)
0927 10:57:13.719997 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.720003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.720186 (+ 183us) server_negotiation.cc:407] Connection header received
0927 10:57:13.720350 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.720352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.720429 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.720530 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:13.720774 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.721254 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.722075 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.722282 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.722384 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.722726 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:13.723023 (+ 297us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:13.723144 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.723203 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.723249 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:57:13.723975 15484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:13.727187 15484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:13.734386 15606 tablet_service.cc:1467] Processing CreateTablet for tablet ceb9ca69f8f5403c867d8950de90d25d (DEFAULT_TABLE table=test-table [id=c9097bfea8774b05870e4956cd7bbeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:13.734987 15606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb9ca69f8f5403c867d8950de90d25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:13.737607 15564 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap starting.
I20260927 10:57:13.738456 15564 tablet_bootstrap.cc:654] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:13.738498 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.733408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:45929 (local address 127.0.0.1:49484)
0927 10:57:13.733679 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.733692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.733772 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.734469 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.734472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.734488 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.734595 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.734603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.735645 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.735648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.736963 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.736972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.738047 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.738051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.738138 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.738180 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.738256 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.738326 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":72,"thread_start_us":135,"threads_started":1}
I20260927 10:57:13.738969 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.733564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.1.46.194:45929)
0927 10:57:13.734009 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.734019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.734036 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:57:13.734135 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.734140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.734322 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.734475 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.734691 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.735493 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.737091 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.737916 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.738023 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.738120 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.738675 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.738789 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":85,"threads_started":1}
I20260927 10:57:13.739837 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.733090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:42877 (local address 127.0.0.1:46662)
0927 10:57:13.733172 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.733186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.733255 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.734124 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.734127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.734144 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.734219 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.734224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.735567 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.735571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.736962 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.736972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.739433 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.739439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.739561 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.739594 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.739648 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.739729 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":97}
I20260927 10:57:13.739995 15564 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: No bootstrap required, opened a new log
I20260927 10:57:13.740074 15564 ts_tablet_manager.cc:1397] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:57:13.740454 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.733226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.1.46.195:42877)
0927 10:57:13.733642 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.733653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.733673 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:57:13.733742 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.733748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.733953 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.734123 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.734328 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.735416 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.738166 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.739274 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.740093 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.740210 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.740255 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.740332 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":140,"threads_started":1}
I20260927 10:57:13.740461 15564 raft_consensus.cc:348] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.740589 15564 raft_consensus.cc:374] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:13.740630 15564 raft_consensus.cc:729] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d326baf6ca4b049bd0ec1329f149ed, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.740746 15564 consensus_queue.cc:260] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.741129 15564 ts_tablet_manager.cc:1428] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:57:13.741873 15744 tablet_service.cc:1467] Processing CreateTablet for tablet ceb9ca69f8f5403c867d8950de90d25d (DEFAULT_TABLE table=test-table [id=c9097bfea8774b05870e4956cd7bbeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:13.743052 15744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb9ca69f8f5403c867d8950de90d25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:13.742980 15876 tablet_service.cc:1467] Processing CreateTablet for tablet ceb9ca69f8f5403c867d8950de90d25d (DEFAULT_TABLE table=test-table [id=c9097bfea8774b05870e4956cd7bbeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:13.743911 15876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb9ca69f8f5403c867d8950de90d25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:13.750800 15955 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Bootstrap starting.
I20260927 10:57:13.751507 15956 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Bootstrap starting.
I20260927 10:57:13.753706 15955 tablet_bootstrap.cc:654] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:13.753895 15956 tablet_bootstrap.cc:654] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:13.754595 15956 log.cc:824] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.754593 15955 log.cc:824] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:13.756153 15955 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: No bootstrap required, opened a new log
I20260927 10:57:13.756165 15956 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: No bootstrap required, opened a new log
I20260927 10:57:13.756342 15955 ts_tablet_manager.cc:1397] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:13.756358 15956 ts_tablet_manager.cc:1397] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:13.759265 15956 raft_consensus.cc:348] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.759490 15956 raft_consensus.cc:374] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:13.759546 15956 raft_consensus.cc:729] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4154861fa94c7bbe592e4ca2382b8b, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.759352 15955 raft_consensus.cc:348] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.759644 15955 raft_consensus.cc:374] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:13.759720 15955 raft_consensus.cc:729] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d1bf633f614d2fb698473974f2a671, State: Initialized, Role: FOLLOWER
I20260927 10:57:13.759985 15956 consensus_queue.cc:260] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.760151 15955 consensus_queue.cc:260] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.760545 15942 heartbeater.cc:503] Master 127.1.46.254:45349 was elected leader, sending a full tablet report...
I20260927 10:57:13.760771 15810 heartbeater.cc:503] Master 127.1.46.254:45349 was elected leader, sending a full tablet report...
I20260927 10:57:13.760874 15956 ts_tablet_manager.cc:1428] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 10:57:13.761039 15955 ts_tablet_manager.cc:1428] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:57:13.762437 15811 tablet.cc:2369] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:57:13.822646 15673 tablet.cc:2369] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:13.833411 15678 raft_consensus.cc:482] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:13.833652 15678 raft_consensus.cc:504] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.834782 15678 leader_election.cc:288] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
I20260927 10:57:13.839313 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.835446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:43697 (local address 127.1.46.194:45929)
0927 10:57:13.835564 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.835568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.835585 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:57:13.835713 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.835715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.835753 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.835843 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.836085 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.836757 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.838122 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.838702 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.838809 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.838854 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.839135 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.839196 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:57:13.839313 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.835134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:41623 (local address 127.1.46.195:42877)
0927 10:57:13.835282 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.835286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.835303 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:57:13.835386 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.835390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.835442 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.835532 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:13.835855 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.836624 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.838122 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.838707 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.838808 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.838854 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.839135 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.839196 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:57:13.839398 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.835031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:42877 (local address 127.1.46.193:41623)
0927 10:57:13.835154 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.835171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.835267 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.835579 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.835584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.835605 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.835744 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.835750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.836734 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.836736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.837999 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.838009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.838814 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.838819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.838972 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.839033 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.839108 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.839237 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":74}
I20260927 10:57:13.839416 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.835303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:45929 (local address 127.1.46.193:43697)
0927 10:57:13.835521 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.835536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:13.835618 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:13.835873 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:13.835875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:13.835890 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:13.835983 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.835987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.836856 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.836859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.838002 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:13.838009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.838823 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.838825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:13.838975 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.839039 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 10:57:13.839108 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:57:13.839237 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":103,"spinlock_wait_cycles":1280,"thread_start_us":90,"threads_started":1}
I20260927 10:57:13.840070 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb9ca69f8f5403c867d8950de90d25d" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" is_pre_election: true
I20260927 10:57:13.840095 15764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb9ca69f8f5403c867d8950de90d25d" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d1bf633f614d2fb698473974f2a671" is_pre_election: true
I20260927 10:57:13.840368 15764 raft_consensus.cc:2454] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 0.
I20260927 10:57:13.840369 15896 raft_consensus.cc:2454] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 0.
I20260927 10:57:13.840790 15558 leader_election.cc:302] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed, 58d1bf633f614d2fb698473974f2a671; no voters:
I20260927 10:57:13.841068 15678 raft_consensus.cc:2781] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:13.841176 15678 raft_consensus.cc:482] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:13.841220 15678 raft_consensus.cc:3037] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:13.842229 15678 raft_consensus.cc:504] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.842609 15678 leader_election.cc:288] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 election: Requested vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
I20260927 10:57:13.843027 15764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb9ca69f8f5403c867d8950de90d25d" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d1bf633f614d2fb698473974f2a671"
I20260927 10:57:13.843147 15764 raft_consensus.cc:3037] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:13.843286 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb9ca69f8f5403c867d8950de90d25d" candidate_uuid: "33d326baf6ca4b049bd0ec1329f149ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4154861fa94c7bbe592e4ca2382b8b"
I20260927 10:57:13.843420 15896 raft_consensus.cc:3037] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:13.844239 15764 raft_consensus.cc:2454] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 1.
I20260927 10:57:13.844448 15896 raft_consensus.cc:2454] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d326baf6ca4b049bd0ec1329f149ed in term 1.
I20260927 10:57:13.844502 15558 leader_election.cc:302] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed, 58d1bf633f614d2fb698473974f2a671; no voters:
I20260927 10:57:13.844696 15678 raft_consensus.cc:2781] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:13.844933 15678 raft_consensus.cc:686] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 LEADER]: Becoming Leader. State: Replica: 33d326baf6ca4b049bd0ec1329f149ed, State: Running, Role: LEADER
I20260927 10:57:13.845186 15678 consensus_queue.cc:237] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:13.848022 15483 catalog_manager.cc:5495] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d326baf6ca4b049bd0ec1329f149ed (127.1.46.193). New cstate: current_term: 1 leader_uuid: "33d326baf6ca4b049bd0ec1329f149ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260927 10:57:13.861322 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.857606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.1.46.193:45671)
0927 10:57:13.857752 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:13.857756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:13.857845 (+ 89us) server_negotiation.cc:407] Connection header received
0927 10:57:13.858025 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:13.858029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:13.858090 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:13.858198 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:13.858433 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.859076 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.859899 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:13.860153 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:13.860262 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:13.860518 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:13.860934 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:13.861065 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:13.861130 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 10:57:13.861184 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 10:57:13.864926 15896 raft_consensus.cc:1261] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Refusing update from remote peer 33d326baf6ca4b049bd0ec1329f149ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:13.864979 15764 raft_consensus.cc:1261] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Refusing update from remote peer 33d326baf6ca4b049bd0ec1329f149ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:13.865772 15962 consensus_queue.cc:1035] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:13.866070 15678 consensus_queue.cc:1035] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:13.873359 15965 mvcc.cc:205] Tried to move back new op lower bound from 7333915172304994304 to 7333915172232310784. Current Snapshot: MvccSnapshot[applied={T|T < 7333915172304994304}]
I20260927 10:57:13.885154 15482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:13.885321 15482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:13.887789 15482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Sending DeleteTablet for 3 replicas of tablet ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:13.888396 15876 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb9ca69f8f5403c867d8950de90d25d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:13 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:46662
I20260927 10:57:13.888537 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 15522
I20260927 10:57:13.888545 15744 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb9ca69f8f5403c867d8950de90d25d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:13 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:49484
I20260927 10:57:13.889623 15975 tablet_replica.cc:330] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: stopping tablet replica
I20260927 10:57:13.889943 15975 raft_consensus.cc:2229] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:13.890228 15975 raft_consensus.cc:2258] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:13.889355 15974 tablet_replica.cc:330] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: stopping tablet replica
I20260927 10:57:13.890744 15974 raft_consensus.cc:2229] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:13.891012 15974 raft_consensus.cc:2258] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:13.891839 15975 ts_tablet_manager.cc:1905] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:13.892251 15974 ts_tablet_manager.cc:1905] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:13.894282 15974 ts_tablet_manager.cc:1918] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:13.894416 15974 log.cc:1196] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:13.894768 15974 ts_tablet_manager.cc:1939] T ceb9ca69f8f5403c867d8950de90d25d P bf4154861fa94c7bbe592e4ca2382b8b: Deleting consensus metadata
I20260927 10:57:13.895165 15975 ts_tablet_manager.cc:1918] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:13.895273 15975 log.cc:1196] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:13.895604 15975 ts_tablet_manager.cc:1939] T ceb9ca69f8f5403c867d8950de90d25d P 58d1bf633f614d2fb698473974f2a671: Deleting consensus metadata
I20260927 10:57:13.895877 15468 catalog_manager.cc:4841] TS bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877): tablet ceb9ca69f8f5403c867d8950de90d25d (table test-table [id=c9097bfea8774b05870e4956cd7bbeb2]) successfully deleted
W20260927 10:57:13.896430 15468 connection.cc:535] client connection to 127.1.46.193:45671 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 10:57:13.896617 15468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:57:13.896711 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 15677
I20260927 10:57:13.897194 15468 catalog_manager.cc:4841] TS 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929): tablet ceb9ca69f8f5403c867d8950de90d25d (table test-table [id=c9097bfea8774b05870e4956cd7bbeb2]) successfully deleted
I20260927 10:57:13.898046 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:13.897782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45671 (local address 127.0.0.1:35582)
0927 10:57:13.897905 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:13.897980 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:57:13.898247 15468 catalog_manager.cc:4584] TS 33d326baf6ca4b049bd0ec1329f149ed (127.1.46.193:45671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ceb9ca69f8f5403c867d8950de90d25d: Network error: Client connection negotiation failed: client connection to 127.1.46.193:45671: connect: Connection refused (error 111)
I20260927 10:57:13.903103 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 15813
I20260927 10:57:13.908578 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 15451
I20260927 10:57:13.914844 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45349
--webserver_interface=127.1.46.254
--webserver_port=43731
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:14.024313 15977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:14.024559 15977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:14.024595 15977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:14.028581 15977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:14.028633 15977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:14.028653 15977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:14.028707 15977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:14.028739 15977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:14.033792 15977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:14.034823 15977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:14.035795 15977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:14.043846 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.043866 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.043855 15985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:14.044020 15977 server_base.cc:1034] running on GCE node
I20260927 10:57:14.044690 15977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:14.045693 15977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:14.046849 15977 hybrid_clock.cc:648] HybridClock initialized: now 1790506634046825 us; error 32 us; skew 500 ppm
Sep 27 10:57:14 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:14.049556 15977 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:14.050504 15977 webserver.cc:460] Webserver started at http://127.1.46.254:43731/ using document root <none> and password file <none>
I20260927 10:57:14.051074 15977 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:14.051141 15977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:14.054731 15977 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:57:14.056665 15992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.057945 15977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:14.058097 15977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01"
format_stamp: "Formatted at 2026-09-27 10:57:11 on dist-test-slave-cffh"
server_key: "3575ed6625e5f64dcab37411eff85acf"
server_key_iv: "df00034c3edf119d63ec9c91861913af"
server_key_version: "encryptionkey@0"
I20260927 10:57:14.058653 15977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:14.081655 15977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:14.084887 15977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:14.085034 15977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:14.092247 15977 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45349
I20260927 10:57:14.092569 16044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45349 every 8 connection(s)
I20260927 10:57:14.093322 15977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:14.097461 16045 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:57:14.099016 16045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Bootstrap starting.
I20260927 10:57:14.100558 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 15977
I20260927 10:57:14.101022 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:45671
--local_ip_for_outbound_sockets=127.1.46.193
--tserver_master_addrs=127.1.46.254:45349
--webserver_port=37133
--webserver_interface=127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:14.116559 16045 log.cc:824] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:14.127686 16045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:57:14.128060 16045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01: Bootstrap complete.
I20260927 10:57:14.132069 16045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:14.132436 16045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc7fbd541ef4c34989e9d2f88ab7a01, State: Initialized, Role: FOLLOWER
I20260927 10:57:14.132910 16045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:14.133016 16045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:14.133104 16045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:14.133224 16045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:57:14.135105 16045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:14.135430 16045 leader_election.cc:302] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fc7fbd541ef4c34989e9d2f88ab7a01; no voters:
I20260927 10:57:14.135778 16045 leader_election.cc:288] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:57:14.135856 16049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:57:14.136036 16049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [term 3 LEADER]: Becoming Leader. State: Replica: 8fc7fbd541ef4c34989e9d2f88ab7a01, State: Running, Role: LEADER
I20260927 10:57:14.136317 16049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } }
I20260927 10:57:14.136960 16045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:14.137219 16049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fc7fbd541ef4c34989e9d2f88ab7a01. Latest consensus state: current_term: 3 leader_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } } }
I20260927 10:57:14.137331 16049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:14.137677 16053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:14.137822 16050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc7fbd541ef4c34989e9d2f88ab7a01" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45349 } } }
I20260927 10:57:14.138000 16050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc7fbd541ef4c34989e9d2f88ab7a01 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:14.142545 16053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:14.143199 16053 catalog_manager.cc:1259] Loaded cluster ID: 0566fe982117408c984f4a3ce03cbe16
I20260927 10:57:14.143277 16053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 10:57:14.144285 16053 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe0cc9ad9ad google::LogMessage::Fail() at ??:0
@ 0x7fe0cc9b088a google::LogMessage::SendToLog() at ??:0
@ 0x7fe0cc9ad4de google::LogMessage::Flush() at ??:0
@ 0x7fe0cc9aea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe0d72ecfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe0d72edd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe0d72ea579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe0d732e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe0cebff3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe0cebffc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe0cec013f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe0cebf0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe0d02346db start_thread at ??:0
@ 0x7fe0cb4ab61f clone at ??:0
W20260927 10:57:14.251314 16047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:14.251677 16047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:14.251762 16047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:14.259054 16047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:14.259148 16047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:14.259281 16047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:57:14.267695 16047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:45671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=37133
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:14.269060 16047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:14.270419 16047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:14.282138 16087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.282195 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.282220 16086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:14.282527 16047 server_base.cc:1034] running on GCE node
I20260927 10:57:14.283066 16047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:14.283847 16047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:14.285100 16047 hybrid_clock.cc:648] HybridClock initialized: now 1790506634285081 us; error 35 us; skew 500 ppm
Sep 27 10:57:14 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:14.288383 16047 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:57:14.289233 16047 webserver.cc:460] Webserver started at http://127.1.46.193:37133/ using document root <none> and password file <none>
I20260927 10:57:14.289776 16047 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:14.289836 16047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:14.293705 16047 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:14.295974 16096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.297109 16047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:14.297219 16047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "33d326baf6ca4b049bd0ec1329f149ed"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "5021a08e9ebaec607331dc779337cd8d"
server_key_iv: "e0949ce613b4aab111e5a342115a3602"
server_key_version: "encryptionkey@0"
I20260927 10:57:14.297777 16047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:14.320248 16047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:14.323413 16047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:14.323603 16047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:14.324156 16047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:14.325192 16103 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:57:14.330281 16047 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:57:14.330358 16047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 10:57:14.330435 16047 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:57:14.331998 16047 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:57:14.332077 16047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:57:14.332351 16103 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap starting.
I20260927 10:57:14.342628 16047 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:45671
I20260927 10:57:14.342849 16210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:45671 every 8 connection(s)
I20260927 10:57:14.343550 16047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:14.348351 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 16047
I20260927 10:57:14.348754 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:45929
--local_ip_for_outbound_sockets=127.1.46.194
--tserver_master_addrs=127.1.46.254:45349
--webserver_port=35253
--webserver_interface=127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:14.353781 16103 log.cc:824] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:14.367388 16103 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:57:14.367803 16103 tablet_bootstrap.cc:492] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Bootstrap complete.
I20260927 10:57:14.369390 16103 ts_tablet_manager.cc:1397] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.007s
I20260927 10:57:14.373085 16103 raft_consensus.cc:348] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:14.373697 16103 raft_consensus.cc:729] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d326baf6ca4b049bd0ec1329f149ed, State: Initialized, Role: FOLLOWER
I20260927 10:57:14.374280 16103 consensus_queue.cc:260] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:14.375456 16103 ts_tablet_manager.cc:1428] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260927 10:57:14.415109 16213 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:14.345294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:58075)
0927 10:57:14.345806 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.345855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:14.346734 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:14.414157 (+ 67423us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":169,"threads_started":1}
W20260927 10:57:14.415583 16211 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:45349 (0 consecutive failures): Network error: Failed to ping master at 127.1.46.254:45349: Client connection negotiation failed: client connection to 127.1.46.254:45349: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:57:14.416298 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.416039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:40757)
0927 10:57:14.416172 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.416240 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 10:57:14.416931 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.416722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:43759)
0927 10:57:14.416823 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.416877 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:57:14.417112 16211 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:57:14.463723 16214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:14.463934 16214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:14.463973 16214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:14.467875 16214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:14.467927 16214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:14.467978 16214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:57:14.473165 16214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:45929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=35253
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:14.474164 16214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:14.475126 16214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:14.484263 16222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.484264 16224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.484311 16221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:14.484359 16214 server_base.cc:1034] running on GCE node
I20260927 10:57:14.484899 16214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:14.485479 16214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:14.486649 16214 hybrid_clock.cc:648] HybridClock initialized: now 1790506634486623 us; error 49 us; skew 500 ppm
Sep 27 10:57:14 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:14.489195 16214 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:57:14.490036 16214 webserver.cc:460] Webserver started at http://127.1.46.194:35253/ using document root <none> and password file <none>
I20260927 10:57:14.490624 16214 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:14.490727 16214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:14.494233 16214 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:14.496244 16231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.497359 16214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:57:14.497486 16214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "58d1bf633f614d2fb698473974f2a671"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "4e3a86b56665ef975ed40dbbdb17d69b"
server_key_iv: "af20cbfb4ab726dd4a523a9ff8860395"
server_key_version: "encryptionkey@0"
I20260927 10:57:14.497998 16214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:14.519586 16214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:14.522342 16214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:14.522527 16214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:14.523077 16214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:14.523815 16214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:14.523885 16214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.523952 16214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:14.523993 16214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.532859 16214 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:45929
I20260927 10:57:14.532891 16344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:45929 every 8 connection(s)
I20260927 10:57:14.533847 16214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:14.536669 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 16214
I20260927 10:57:14.537205 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:42877
--local_ip_for_outbound_sockets=127.1.46.195
--tserver_master_addrs=127.1.46.254:45349
--webserver_port=38659
--webserver_interface=127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:14.538122 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.536030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:56209)
0927 10:57:14.536523 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.537183 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
W20260927 10:57:14.538628 16345 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:45349 (0 consecutive failures): Network error: Failed to ping master at 127.1.46.254:45349: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
I20260927 10:57:14.539227 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.538953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:40671)
0927 10:57:14.539093 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.539154 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:57:14.540134 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.539824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:34375)
0927 10:57:14.539957 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.540056 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 10:57:14.540334 16345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:57:14.648103 16348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:14.648355 16348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:14.648443 16348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:14.652530 16348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:14.652583 16348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:14.652702 16348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:57:14.658000 16348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:42877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=38659
--tserver_master_addrs=127.1.46.254:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:14.659013 16348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:14.660000 16348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:14.669265 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.669265 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:14.669330 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:14.669425 16348 server_base.cc:1034] running on GCE node
I20260927 10:57:14.670027 16348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:14.670706 16348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:14.671895 16348 hybrid_clock.cc:648] HybridClock initialized: now 1790506634671860 us; error 53 us; skew 500 ppm
Sep 27 10:57:14 dist-test-slave-cffh krb5kdc[14904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:14.674557 16348 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:57:14.675396 16348 webserver.cc:460] Webserver started at http://127.1.46.195:38659/ using document root <none> and password file <none>
I20260927 10:57:14.675940 16348 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:14.676029 16348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:14.679598 16348 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 10:57:14.681499 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.682726 16348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:14.682843 16348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "bf4154861fa94c7bbe592e4ca2382b8b"
format_stamp: "Formatted at 2026-09-27 10:57:12 on dist-test-slave-cffh"
server_key: "6970fccbe4276754f9f0ca94ab7a5dbc"
server_key_iv: "320a850fa8bded407296422bca748b74"
server_key_version: "encryptionkey@0"
I20260927 10:57:14.683344 16348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:14.703151 16348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:14.705977 16348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:14.706136 16348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:14.706717 16348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:14.707501 16348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:14.707551 16348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.707593 16348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:14.707619 16348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:14.717290 16348 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:42877
I20260927 10:57:14.717301 16476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:42877 every 8 connection(s)
I20260927 10:57:14.718245 16348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:14.721932 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.720341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:37567)
0927 10:57:14.720784 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.721272 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
W20260927 10:57:14.722426 16477 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:45349 (0 consecutive failures): Network error: Failed to ping master at 127.1.46.254:45349: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
I20260927 10:57:14.723001 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.722763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:39259)
0927 10:57:14.722874 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.722929 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:57:14.723620 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:14.723428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:54859)
0927 10:57:14.723526 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:14.723571 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:57:14.723837 16477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:57:14.724013 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 16348
I20260927 10:57:15.418624 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.417895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:52117)
0927 10:57:15.418364 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:15.418436 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":214,"threads_started":1}
I20260927 10:57:15.541115 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.540785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:59833)
0927 10:57:15.540983 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:15.541028 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20260927 10:57:15.724692 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:15.724318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:44659)
0927 10:57:15.724531 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:15.724586 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
I20260927 10:57:16.247483 16484 raft_consensus.cc:482] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:16.247727 16484 raft_consensus.cc:504] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:16.248832 16484 leader_election.cc:288] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
Sep 27 10:57:16 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.194@KRBTEST.COM
Sep 27 10:57:16 dist-test-slave-cffh krb5kdc[14904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.195@KRBTEST.COM
I20260927 10:57:16.266875 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.249219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:46515 (local address 127.1.46.195:42877)
0927 10:57:16.249633 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.249660 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.249705 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:57:16.250409 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.250415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.250832 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.251086 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:16.252410 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.253484 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.254728 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.255209 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.255382 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.262385 (+ 7003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:16.262390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:16.262401 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:16.262430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:16.264297 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.264837 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.264842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.264846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.264959 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.265330 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.265335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.265338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.265731 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:16.265868 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.266390 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.266665 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":66,"threads_started":1}
I20260927 10:57:16.266908 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.249429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:45929 (local address 127.1.46.193:49427)
0927 10:57:16.249680 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.249690 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.249787 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.251230 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.251233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.251755 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:16.252228 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.252245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.253664 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.253667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.254628 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.254638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.255321 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.255324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.255539 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.260136 (+ 4597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:16.260154 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:16.261840 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:16.264269 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:16.264276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:16.264290 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:16.264705 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:16.265121 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:16.265124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:16.265125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:16.265234 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:16.265856 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:16.265863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:16.266099 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.266281 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.266668 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":113,"spinlock_wait_cycles":29312,"thread_start_us":70,"threads_started":1}
I20260927 10:57:16.266928 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.249085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:42877 (local address 127.1.46.193:46515)
0927 10:57:16.249321 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.249337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:16.249449 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:16.251188 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:16.251194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:16.251756 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:16.252230 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.252244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.253617 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.253624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.254619 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:16.254629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.255349 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.255352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:16.255568 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.260155 (+ 4587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:16.260167 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:16.262194 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:16.264421 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:16.264424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:16.264425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:16.264707 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:16.265073 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:16.265079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:16.265081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:16.265227 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:16.265858 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:16.265864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:16.266095 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 10:57:16.266281 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 10:57:16.266748 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":326,"spinlock_wait_cycles":14976,"thread_start_us":85,"threads_started":1}
I20260927 10:57:16.266889 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.249611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:49427 (local address 127.1.46.194:45929)
0927 10:57:16.249949 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:16.249958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:16.249986 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:57:16.250669 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:16.250675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:16.250954 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:16.251257 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:16.252409 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.253521 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.254765 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:16.255200 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:16.255356 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:16.262118 (+ 6762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:16.262127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:16.262147 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:16.262180 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:16.264146 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.264841 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.264845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.264849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.264959 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:16.265331 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:16.265336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:16.265339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:16.265731 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:16.265868 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:16.266391 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 10:57:16.266701 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
W20260927 10:57:16.269510 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
W20260927 10:57:16.269713 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:16.269809 16097 leader_election.cc:302] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed; no voters: 58d1bf633f614d2fb698473974f2a671, bf4154861fa94c7bbe592e4ca2382b8b
I20260927 10:57:16.270073 16484 raft_consensus.cc:2726] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:57:16.419691 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.419403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:46287)
0927 10:57:16.419551 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.419605 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:57:16.542188 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.541789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:38753)
0927 10:57:16.542019 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.542072 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260927 10:57:16.725710 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:16.725320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:50265)
0927 10:57:16.725563 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:16.725616 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":66,"threads_started":1}
I20260927 10:57:17.420830 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.420449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:39757)
0927 10:57:17.420681 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.420734 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":65,"threads_started":1}
I20260927 10:57:17.543300 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.542904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:39545)
0927 10:57:17.543147 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.543216 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260927 10:57:17.726933 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:17.726474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:49431)
0927 10:57:17.726783 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:17.726838 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
I20260927 10:57:18.333510 16494 raft_consensus.cc:482] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:18.333657 16494 raft_consensus.cc:504] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:18.334000 16494 leader_election.cc:288] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
W20260927 10:57:18.334900 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
W20260927 10:57:18.335067 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:18.335145 16097 leader_election.cc:302] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed; no voters: 58d1bf633f614d2fb698473974f2a671, bf4154861fa94c7bbe592e4ca2382b8b
I20260927 10:57:18.335325 16494 raft_consensus.cc:2726] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:57:18.421947 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:18.421532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:54579)
0927 10:57:18.421793 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:18.421843 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
I20260927 10:57:18.544282 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:18.543939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:43955)
0927 10:57:18.544151 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:18.544198 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260927 10:57:18.728073 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:18.727651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:37531)
0927 10:57:18.727906 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:18.727963 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
I20260927 10:57:19.423269 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.422843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:57875)
0927 10:57:19.423104 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.423173 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260927 10:57:19.545351 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.544962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:49291)
0927 10:57:19.545200 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.545259 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260927 10:57:19.729387 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:19.728828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:53251)
0927 10:57:19.729088 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:19.729149 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260927 10:57:19.924996 16501 raft_consensus.cc:482] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:19.925117 16501 raft_consensus.cc:504] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:19.925521 16501 leader_election.cc:288] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
W20260927 10:57:19.926290 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
W20260927 10:57:19.926440 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:19.926497 16097 leader_election.cc:302] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed; no voters: 58d1bf633f614d2fb698473974f2a671, bf4154861fa94c7bbe592e4ca2382b8b
I20260927 10:57:19.926718 16501 raft_consensus.cc:2726] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:57:20.424489 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.424008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:59957)
0927 10:57:20.424320 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.424382 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":130,"threads_started":1}
I20260927 10:57:20.546447 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.546052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:47311)
0927 10:57:20.546317 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.546365 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
I20260927 10:57:20.730440 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:20.730058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:46399)
0927 10:57:20.730307 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:20.730357 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260927 10:57:21.425721 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.425274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:46801)
0927 10:57:21.425572 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:21.425628 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260927 10:57:21.547557 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.547165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:35369)
0927 10:57:21.547427 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:21.547477 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260927 10:57:21.731547 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:21.731120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:33935)
0927 10:57:21.731416 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:21.731465 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":84,"threads_started":1}
I20260927 10:57:22.071393 16508 raft_consensus.cc:482] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:22.071527 16508 raft_consensus.cc:504] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:22.071884 16508 leader_election.cc:288] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
W20260927 10:57:22.072724 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
W20260927 10:57:22.072901 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:22.072961 16097 leader_election.cc:302] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed; no voters: 58d1bf633f614d2fb698473974f2a671, bf4154861fa94c7bbe592e4ca2382b8b
I20260927 10:57:22.073117 16508 raft_consensus.cc:2726] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:57:22.426962 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:22.426510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:40487)
0927 10:57:22.426770 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:22.426849 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260927 10:57:22.548593 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:22.548195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:50303)
0927 10:57:22.548434 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:22.548484 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260927 10:57:22.732573 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:22.732213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:56479)
0927 10:57:22.732440 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:22.732488 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260927 10:57:23.428104 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:23.427695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:50159)
0927 10:57:23.427953 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:23.428005 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260927 10:57:23.549587 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:23.549221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:43843)
0927 10:57:23.549454 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:23.549504 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260927 10:57:23.733618 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:23.733213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:52427)
0927 10:57:23.733473 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:23.733529 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260927 10:57:24.429355 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:24.428957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:42001)
0927 10:57:24.429211 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:24.429263 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260927 10:57:24.550608 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:24.550222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:35209)
0927 10:57:24.550451 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:24.550501 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
I20260927 10:57:24.734819 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:24.734332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:54509)
0927 10:57:24.734545 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:24.734684 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260927 10:57:25.430464 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:25.430081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:36787)
0927 10:57:25.430305 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:25.430354 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260927 10:57:25.551680 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:25.551305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:52033)
0927 10:57:25.551530 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:25.551586 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260927 10:57:25.735877 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:25.735508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:37793)
0927 10:57:25.735731 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:25.735784 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260927 10:57:26.431584 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:26.431179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:34443)
0927 10:57:26.431423 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:26.431477 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260927 10:57:26.552791 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:26.552353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:35163)
0927 10:57:26.552649 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:26.552702 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260927 10:57:26.736898 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:26.736528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:46281)
0927 10:57:26.736766 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:26.736814 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260927 10:57:27.432793 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:27.432350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:40333)
0927 10:57:27.432640 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:27.432694 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260927 10:57:27.553854 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:27.553430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:34175)
0927 10:57:27.553700 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:27.553760 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":67,"threads_started":1}
I20260927 10:57:27.737898 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:27.737509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:38663)
0927 10:57:27.737758 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:27.737809 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20260927 10:57:28.399382 16527 raft_consensus.cc:482] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:28.399515 16527 raft_consensus.cc:504] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4154861fa94c7bbe592e4ca2382b8b" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 42877 } } peers { permanent_uuid: "58d1bf633f614d2fb698473974f2a671" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 45929 } } peers { permanent_uuid: "33d326baf6ca4b049bd0ec1329f149ed" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45671 } }
I20260927 10:57:28.400125 16527 leader_election.cc:288] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877), 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929)
W20260927 10:57:28.400868 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf4154861fa94c7bbe592e4ca2382b8b (127.1.46.195:42877): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
W20260927 10:57:28.401016 16097 leader_election.cc:341] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d1bf633f614d2fb698473974f2a671 (127.1.46.194:45929): Not found: Tablet not found: ceb9ca69f8f5403c867d8950de90d25d
I20260927 10:57:28.401090 16097 leader_election.cc:302] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33d326baf6ca4b049bd0ec1329f149ed; no voters: 58d1bf633f614d2fb698473974f2a671, bf4154861fa94c7bbe592e4ca2382b8b
I20260927 10:57:28.401261 16527 raft_consensus.cc:2726] T ceb9ca69f8f5403c867d8950de90d25d P 33d326baf6ca4b049bd0ec1329f149ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:57:28.433822 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:28.433469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:35237)
0927 10:57:28.433688 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:28.433737 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260927 10:57:28.554955 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:28.554527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:39713)
0927 10:57:28.554824 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:28.554872 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":150,"threads_started":1}
I20260927 10:57:28.739065 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:28.738665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:37161)
0927 10:57:28.738921 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:28.738970 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":111,"threads_started":1}
I20260927 10:57:29.435194 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:29.434696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.193:42433)
0927 10:57:29.434987 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:29.435064 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
I20260927 10:57:29.556361 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:29.555863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.194:50067)
0927 10:57:29.556156 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:29.556248 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":114,"threads_started":1}
I20260927 10:57:29.728907 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 16047
I20260927 10:57:29.734669 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 16214
I20260927 10:57:29.740087 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 16348
I20260927 10:57:29.740115 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:29.739761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45349 (local address 127.1.46.195:39951)
0927 10:57:29.739973 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:29.740018 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
2026-09-27T10:57:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19014 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:57:29 dist-test-slave-cffh krb5kdc[16536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:57:29 dist-test-slave-cffh krb5kdc[16536](info): set up 2 sockets
Sep 27 10:57:29 dist-test-slave-cffh krb5kdc[16536](info): commencing operation
krb5kdc: starting...
W20260927 10:57:30.767218 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:57:30 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:57:30.878417 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40117
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:30.985739 16549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:30.986008 16549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:30.986079 16549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:30.990075 16549 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:57:30.990154 16549 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:57:30.990185 16549 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:57:30.990218 16549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:30.990257 16549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:30.990311 16549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:30.990355 16549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:30.990391 16549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:30.995628 16549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:30.996611 16549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:30.997583 16549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:31.006264 16555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.006314 16549 server_base.cc:1034] running on GCE node
W20260927 10:57:31.006430 16557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:31.006685 16554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.007236 16549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:31.008266 16549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:31.009464 16549 hybrid_clock.cc:648] HybridClock initialized: now 1790506651009428 us; error 53 us; skew 500 ppm
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:31.012241 16549 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:31.013082 16549 webserver.cc:460] Webserver started at http://127.1.46.254:38753/ using document root <none> and password file <none>
I20260927 10:57:31.013609 16549 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:31.013691 16549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:31.013901 16549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:31.015939 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "8eff24e0d9cb47ad951a506657c11258"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "2bd6ae78c1403df906398b4932fb4877"
server_key_iv: "5774a9e83573189e0a9c4a6ed9db0e39"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.016482 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "8eff24e0d9cb47ad951a506657c11258"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "2bd6ae78c1403df906398b4932fb4877"
server_key_iv: "5774a9e83573189e0a9c4a6ed9db0e39"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.019851 16549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:31.022107 16564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.023353 16549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:31.023481 16549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8eff24e0d9cb47ad951a506657c11258"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "2bd6ae78c1403df906398b4932fb4877"
server_key_iv: "5774a9e83573189e0a9c4a6ed9db0e39"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.023578 16549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:31.036388 16549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:31.039060 16549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:31.039250 16549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:31.045819 16549 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:40117
I20260927 10:57:31.046115 16616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:40117 every 8 connection(s)
I20260927 10:57:31.046730 16549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:31.049297 16617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:31.053668 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 16549
I20260927 10:57:31.053807 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:57:31.054082 1211 external_mini_cluster.cc:1428] Setting key 01fc8452eb6a17d32c13a16318d1625d
I20260927 10:57:31.054559 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Bootstrap starting.
I20260927 10:57:31.056644 16617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:31.057518 16617 log.cc:824] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:31.059468 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: No bootstrap required, opened a new log
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:31.063638 16617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:31.063827 16617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:31.063898 16617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eff24e0d9cb47ad951a506657c11258, State: Initialized, Role: FOLLOWER
I20260927 10:57:31.064363 16617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:31.064512 16617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:31.064575 16617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:31.064669 16617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:31.065490 16617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:31.065795 16617 leader_election.cc:302] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eff24e0d9cb47ad951a506657c11258; no voters:
I20260927 10:57:31.066068 16617 leader_election.cc:288] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:57:31.066380 16623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:31.066561 16623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 LEADER]: Becoming Leader. State: Replica: 8eff24e0d9cb47ad951a506657c11258, State: Running, Role: LEADER
I20260927 10:57:31.066949 16623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:31.067216 16617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:31.068450 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eff24e0d9cb47ad951a506657c11258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } } }
I20260927 10:57:31.068560 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:31.068764 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.055402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.1.46.254:40117)
0927 10:57:31.055765 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.055786 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.055824 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:57:31.056394 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.056408 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.056729 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.056991 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:31.057894 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.059033 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.060167 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.060585 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.060708 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.063349 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:31.063353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:31.063365 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:31.063388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:31.065266 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.065702 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.065709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.065715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.065821 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.066093 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.066104 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.066107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.066625 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:31.067363 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.067499 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.067771 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":130,"threads_started":1}
I20260927 10:57:31.068785 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eff24e0d9cb47ad951a506657c11258. Latest consensus state: current_term: 1 leader_uuid: "8eff24e0d9cb47ad951a506657c11258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } } }
I20260927 10:57:31.068849 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:31.069738 16631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:31.072129 16631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:31.076869 16631 catalog_manager.cc:1347] Generated new cluster ID: 0ba30133a5c54afbb915c755a21befd8
I20260927 10:57:31.076928 16631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:31.092883 16631 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:57:31.093779 16631 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:31.100968 16631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Generated new TSK 0
I20260927 10:57:31.101598 16631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:57:31.129120 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40117
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:31.235210 16643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:31.235435 16643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:31.235476 16643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:31.239360 16643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:31.239405 16643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:31.239459 16643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:57:31.244369 16643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:31.245208 16643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:31.246119 16643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:31.255013 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:31.254992 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.255055 16643 server_base.cc:1034] running on GCE node
W20260927 10:57:31.255146 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.255581 16643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:31.256150 16643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:31.257306 16643 hybrid_clock.cc:648] HybridClock initialized: now 1790506651257293 us; error 25 us; skew 500 ppm
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:31.259941 16643 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:57:31.260668 16643 webserver.cc:460] Webserver started at http://127.1.46.193:39021/ using document root <none> and password file <none>
I20260927 10:57:31.261142 16643 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:31.261196 16643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:31.261343 16643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:31.263227 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "6f36746b42d64e579a5404c69686c137"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "6ed167f7cd68cd2e4e182d2c39a5f6e5"
server_key_iv: "f79829757dee3117eb0cf7d4dfa38da2"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.263710 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "6f36746b42d64e579a5404c69686c137"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "6ed167f7cd68cd2e4e182d2c39a5f6e5"
server_key_iv: "f79829757dee3117eb0cf7d4dfa38da2"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.267145 16643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:31.269430 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.270689 16643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:31.270793 16643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "6f36746b42d64e579a5404c69686c137"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "6ed167f7cd68cd2e4e182d2c39a5f6e5"
server_key_iv: "f79829757dee3117eb0cf7d4dfa38da2"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.270887 16643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:31.290321 16643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:31.293361 16643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:31.293547 16643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:31.294111 16643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:31.294868 16643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:31.294934 16643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.295006 16643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:31.295045 16643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.303378 16643 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:39575
I20260927 10:57:31.303413 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:39575 every 8 connection(s)
I20260927 10:57:31.304188 16643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:31.305696 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 16643
I20260927 10:57:31.305840 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:57:31.306236 1211 external_mini_cluster.cc:1428] Setting key 44fb4ddde742e70464320706138fdccf
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:57:31.318842 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.305883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:60873 (local address 127.1.46.254:40117)
0927 10:57:31.306140 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.306146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.307380 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 10:57:31.308417 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.308423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.308480 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.308571 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:31.309826 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.310337 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.312210 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.312426 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.312504 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.315332 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:31.315340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:31.315355 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:31.315377 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:31.316794 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.317269 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.317273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.317275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.317353 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.317727 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.317731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.317733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.317894 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:31.317992 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.318492 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.318677 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:57:31.319655 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.306342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:60873)
0927 10:57:31.307211 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.307265 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.308204 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.308691 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.308697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.309039 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:31.309670 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.309692 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.310463 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.310469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.312079 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.312095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.312587 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.312592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.312827 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.313440 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:31.313468 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:31.315181 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:31.316904 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:31.316910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:31.316923 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:31.317168 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:31.317460 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:31.317464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:31.317467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:31.317642 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:31.317990 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:31.317996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:31.318240 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.318409 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.318724 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":170,"threads_started":1}
I20260927 10:57:31.320819 16772 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40117
I20260927 10:57:31.321063 16772 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:31.321525 16772 heartbeater.cc:511] Master 127.1.46.254:40117 requested a full tablet report, sending...
I20260927 10:57:31.323144 16580 ts_manager.cc:194] Registered new tserver with Master: 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575)
I20260927 10:57:31.324922 16580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:60873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:57:31.335503 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40117
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:31.445268 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:31.445500 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:31.445551 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:31.449733 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:31.449790 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:31.449849 16777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:57:31.454751 16777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:31.455583 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:31.456439 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:31.465575 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:31.465564 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.465631 16777 server_base.cc:1034] running on GCE node
W20260927 10:57:31.465566 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.466224 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:31.466785 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:31.467928 16777 hybrid_clock.cc:648] HybridClock initialized: now 1790506651467900 us; error 39 us; skew 500 ppm
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:31.470920 16777 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:57:31.471673 16777 webserver.cc:460] Webserver started at http://127.1.46.194:38779/ using document root <none> and password file <none>
I20260927 10:57:31.472193 16777 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:31.472263 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:31.472487 16777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:31.474395 16777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "092d377138004a2aa121f50b81bc3c68"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "dae24c501c74614bd90d14df1816e98a"
server_key_iv: "f68e06636caec9cffd0c32f5253d8d78"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.474982 16777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "092d377138004a2aa121f50b81bc3c68"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "dae24c501c74614bd90d14df1816e98a"
server_key_iv: "f68e06636caec9cffd0c32f5253d8d78"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.478276 16777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:57:31.480760 16792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.481909 16777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:31.482059 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "092d377138004a2aa121f50b81bc3c68"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "dae24c501c74614bd90d14df1816e98a"
server_key_iv: "f68e06636caec9cffd0c32f5253d8d78"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.482168 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:31.496132 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:31.499262 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:31.499450 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:31.500001 16777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:31.500803 16777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:31.500875 16777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.500942 16777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:31.500994 16777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.510602 16777 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:40381
I20260927 10:57:31.510685 16905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:40381 every 8 connection(s)
I20260927 10:57:31.511528 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:31.521159 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 16777
I20260927 10:57:31.521307 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:57:31.521664 1211 external_mini_cluster.cc:1428] Setting key f0c8667a365e4b61f3273ef5323cc3a0
I20260927 10:57:31.525228 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.513310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:38301 (local address 127.1.46.254:40117)
0927 10:57:31.513471 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.513476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.514528 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 10:57:31.515444 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.515447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.515495 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.515585 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:31.516684 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.517231 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.518352 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.518670 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.518810 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.521200 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:31.521207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:31.521213 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:31.521245 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:31.522977 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.523560 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.523565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.523567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.523666 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.523995 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.524002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.524006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.524277 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:31.524395 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.524852 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.524995 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:57:31.526275 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.513726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:38301)
0927 10:57:31.514370 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.514406 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.515247 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.515749 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.515755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.516095 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:31.516536 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.516548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.517350 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.517353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.518236 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.518245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.518845 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.518849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.519032 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.519559 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:31.519576 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:31.521023 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:31.523096 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:31.523102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:31.523115 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:31.523427 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:31.523766 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:31.523768 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:31.523770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:31.523895 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:31.524390 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:31.524395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:31.524613 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.524762 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.525092 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":127,"threads_started":1}
I20260927 10:57:31.527496 16906 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40117
I20260927 10:57:31.527751 16906 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:31.528348 16906 heartbeater.cc:511] Master 127.1.46.254:40117 requested a full tablet report, sending...
I20260927 10:57:31.529466 16580 ts_manager.cc:194] Registered new tserver with Master: 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381)
I20260927 10:57:31.530323 16580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:38301
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:57:31.549857 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40117
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:31.656497 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:31.656728 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:31.656766 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:31.660776 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:31.660826 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:31.660910 16911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:57:31.665969 16911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:31.666939 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:31.667918 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:31.677477 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.677706 16911 server_base.cc:1034] running on GCE node
W20260927 10:57:31.677475 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:31.677472 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:31.678283 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:31.678920 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:31.680107 16911 hybrid_clock.cc:648] HybridClock initialized: now 1790506651680092 us; error 24 us; skew 500 ppm
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:31.682829 16911 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:57:31.683674 16911 webserver.cc:460] Webserver started at http://127.1.46.195:39343/ using document root <none> and password file <none>
I20260927 10:57:31.684193 16911 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:31.684278 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:31.684490 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:57:31.686383 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "eb4ef2fb65d44ea19ffc86248f824f55"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "cc9c9711f7cde58925b0315850e9f1f9"
server_key_iv: "f4b3ea48a031e0e8d0a8ce98253bd273"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.686957 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "eb4ef2fb65d44ea19ffc86248f824f55"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "cc9c9711f7cde58925b0315850e9f1f9"
server_key_iv: "f4b3ea48a031e0e8d0a8ce98253bd273"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.690253 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:57:31.692468 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.693630 16911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:31.693758 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "eb4ef2fb65d44ea19ffc86248f824f55"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "cc9c9711f7cde58925b0315850e9f1f9"
server_key_iv: "f4b3ea48a031e0e8d0a8ce98253bd273"
server_key_version: "encryptionkey@0"
I20260927 10:57:31.693861 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:31.708690 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:31.712021 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:31.712234 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:31.712785 16911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:31.713550 16911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:31.713615 16911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.713678 16911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:31.713711 16911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:31.722045 16911 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39621
I20260927 10:57:31.722071 17039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39621 every 8 connection(s)
I20260927 10:57:31.722894 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:31.725051 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 16911
I20260927 10:57:31.725196 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:57:31.725467 1211 external_mini_cluster.cc:1428] Setting key e6b6bd3bdde7cfa30f9a1b727ac3dbd3
Sep 27 10:57:31 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506651, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:31.735339 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.724517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:45563 (local address 127.1.46.254:40117)
0927 10:57:31.724677 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.724682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.725943 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 10:57:31.726923 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.726926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.726977 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.727081 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:31.728224 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.728761 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.729860 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.730084 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.730182 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.732447 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:31.732450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:31.732452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:31.732472 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:31.733667 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.734108 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.734110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.734111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.734178 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.734450 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.734452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.734453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.734669 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:31.734744 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.735142 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.735231 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:57:31.736029 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.724887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:45563)
0927 10:57:31.725803 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.725851 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.726748 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.727273 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.727279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.727597 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:31.728097 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.728111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.728870 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.728873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.729731 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.729739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.730216 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.730218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.730348 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.730935 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:31.730950 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:31.732309 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:31.733778 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:31.733782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:31.733791 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:31.734024 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:31.734253 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:31.734255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:31.734256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:31.734377 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:31.734755 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:31.734759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:31.734948 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.735091 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.735300 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":608,"threads_started":1}
I20260927 10:57:31.736994 17040 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40117
I20260927 10:57:31.737190 17040 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:31.737560 17040 heartbeater.cc:511] Master 127.1.46.254:40117 requested a full tablet report, sending...
I20260927 10:57:31.738822 16580 ts_manager.cc:194] Registered new tserver with Master: eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621)
I20260927 10:57:31.739409 16580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:45563
I20260927 10:57:31.750497 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:31.759003 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.752385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.1.46.254:40117)
0927 10:57:31.752487 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.752490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.752708 (+ 218us) server_negotiation.cc:407] Connection header received
0927 10:57:31.752777 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.752779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.752813 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.752875 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:31.753459 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.753932 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.754781 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.754990 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.755060 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.756254 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:31.756257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:31.756258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:31.756278 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:31.757648 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.758013 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.758016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.758017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.758083 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:31.758313 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:31.758315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:31.758316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:31.758489 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:31.758672 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.758823 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.758903 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:57:31.761458 16580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:31.764485 16580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:31.778641 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.772200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39621 (local address 127.0.0.1:38966)
0927 10:57:31.772554 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.772603 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.772708 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.773743 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.773750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.773777 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:31.773930 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.773944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.775464 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.775468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.777120 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.777133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.778202 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.778208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.778301 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.778322 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.778368 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.778443 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":159,"thread_start_us":84,"threads_started":1}
I20260927 10:57:31.778798 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.773089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.1.46.194:40381)
0927 10:57:31.773480 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.773489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.773975 (+ 486us) server_negotiation.cc:407] Connection header received
0927 10:57:31.774081 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.774089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.774271 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.774425 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:31.774701 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.775790 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.777389 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.778184 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.778274 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.778404 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.778528 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.778659 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":117,"threads_started":1}
I20260927 10:57:31.778991 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.772956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.1.46.193:39575)
0927 10:57:31.773320 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.773337 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.773362 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:57:31.773440 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.773448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.773686 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.774162 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:31.774363 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.775301 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.777310 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.777874 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.778017 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.778168 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.778706 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.778823 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":66,"threads_started":1}
I20260927 10:57:31.778644 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.772471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39575 (local address 127.0.0.1:53326)
0927 10:57:31.772918 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.772931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.773011 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.773858 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.773862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.773875 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:31.774006 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.774014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.775448 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.775454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.777206 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.777218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.778023 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.778027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.778140 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.778181 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.778295 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.778412 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":168,"thread_start_us":113,"threads_started":1}
I20260927 10:57:31.779500 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.772922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:40381 (local address 127.0.0.1:47566)
0927 10:57:31.773765 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.773782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.773870 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.774430 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.774433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.774451 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:31.774532 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.774537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.776121 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.776124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.777254 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.777264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.778293 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.778297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.778384 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.778409 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.778498 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.779203 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":158,"thread_start_us":97,"threads_started":1}
I20260927 10:57:31.779773 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.772315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38966 (local address 127.1.46.195:39621)
0927 10:57:31.773257 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.773269 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.773290 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:57:31.773358 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.773365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.773562 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.774078 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:31.774181 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.775310 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.777259 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.778047 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.778993 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.779520 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.779570 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.779649 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":133,"threads_started":1}
I20260927 10:57:31.782249 16706 tablet_service.cc:1467] Processing CreateTablet for tablet 07b2e592c5ce4c9587b6577d98bd41bb (DEFAULT_TABLE table=test-table [id=0f9a8d42e2e24d77b7f8f06312fb693a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:31.782253 16840 tablet_service.cc:1467] Processing CreateTablet for tablet 07b2e592c5ce4c9587b6577d98bd41bb (DEFAULT_TABLE table=test-table [id=0f9a8d42e2e24d77b7f8f06312fb693a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:31.783372 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07b2e592c5ce4c9587b6577d98bd41bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:31.783372 16840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07b2e592c5ce4c9587b6577d98bd41bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:31.784425 16974 tablet_service.cc:1467] Processing CreateTablet for tablet 07b2e592c5ce4c9587b6577d98bd41bb (DEFAULT_TABLE table=test-table [id=0f9a8d42e2e24d77b7f8f06312fb693a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:31.785322 16974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07b2e592c5ce4c9587b6577d98bd41bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:31.790531 17057 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Bootstrap starting.
I20260927 10:57:31.790987 17058 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Bootstrap starting.
I20260927 10:57:31.791643 17059 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Bootstrap starting.
I20260927 10:57:31.793358 17058 tablet_bootstrap.cc:654] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:31.794000 17059 tablet_bootstrap.cc:654] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:31.794000 17057 tablet_bootstrap.cc:654] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:31.794783 17058 log.cc:824] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:31.794860 17057 log.cc:824] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:31.794860 17059 log.cc:824] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:31.796559 17058 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: No bootstrap required, opened a new log
I20260927 10:57:31.796671 17057 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: No bootstrap required, opened a new log
I20260927 10:57:31.796675 17059 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: No bootstrap required, opened a new log
I20260927 10:57:31.796777 17058 ts_tablet_manager.cc:1397] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:31.796842 17059 ts_tablet_manager.cc:1397] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:57:31.796854 17057 ts_tablet_manager.cc:1397] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:57:31.799906 17058 raft_consensus.cc:348] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.800158 17058 raft_consensus.cc:374] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:31.800240 17058 raft_consensus.cc:729] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f36746b42d64e579a5404c69686c137, State: Initialized, Role: FOLLOWER
I20260927 10:57:31.800578 17057 raft_consensus.cc:348] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.800586 17059 raft_consensus.cc:348] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.800827 17059 raft_consensus.cc:374] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:31.800767 17058 consensus_queue.cc:260] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.800894 17059 raft_consensus.cc:729] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4ef2fb65d44ea19ffc86248f824f55, State: Initialized, Role: FOLLOWER
I20260927 10:57:31.801127 17057 raft_consensus.cc:374] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:31.801223 17057 raft_consensus.cc:729] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092d377138004a2aa121f50b81bc3c68, State: Initialized, Role: FOLLOWER
I20260927 10:57:31.801501 16772 heartbeater.cc:503] Master 127.1.46.254:40117 was elected leader, sending a full tablet report...
I20260927 10:57:31.801661 17057 consensus_queue.cc:260] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.801903 17058 ts_tablet_manager.cc:1428] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:57:31.802273 16906 heartbeater.cc:503] Master 127.1.46.254:40117 was elected leader, sending a full tablet report...
I20260927 10:57:31.801301 17059 consensus_queue.cc:260] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.802538 17057 ts_tablet_manager.cc:1428] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:57:31.803066 17040 heartbeater.cc:503] Master 127.1.46.254:40117 was elected leader, sending a full tablet report...
I20260927 10:57:31.803346 17059 ts_tablet_manager.cc:1428] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 10:57:31.805154 16773 tablet.cc:2369] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:31.962244 17064 raft_consensus.cc:482] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:31.962436 17064 raft_consensus.cc:504] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.963585 17064 leader_election.cc:288] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575)
I20260927 10:57:31.967340 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.963870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39575 (local address 127.1.46.194:60223)
0927 10:57:31.964014 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.964032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.964158 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.964467 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.964470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.964488 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:31.964615 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.964620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.965369 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.965375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.966186 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.966194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.966883 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.966887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.967004 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.967029 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.967124 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.967193 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":22}
I20260927 10:57:31.967340 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.964076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:60223 (local address 127.1.46.193:39575)
0927 10:57:31.964235 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.964242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.964256 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:57:31.964310 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.964313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.964352 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.964446 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:31.964753 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.965262 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.966325 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.966767 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.966894 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.966958 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.967140 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.967209 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:57:31.967803 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.963991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:43377 (local address 127.1.46.195:39621)
0927 10:57:31.964131 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:31.964137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:31.964594 (+ 457us) server_negotiation.cc:407] Connection header received
0927 10:57:31.964735 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:31.964739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:31.964790 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:31.964869 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:31.965133 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.965624 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.966711 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.967169 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.967289 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.967354 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:31.967609 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 10:57:31.967679 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:57:31.968012 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b2e592c5ce4c9587b6577d98bd41bb" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f36746b42d64e579a5404c69686c137" is_pre_election: true
I20260927 10:57:31.968212 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:31.963870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39621 (local address 127.1.46.194:43377)
0927 10:57:31.964472 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:31.964483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:31.964604 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:31.964878 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:31.964880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:31.964895 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:31.965010 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.965016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.965737 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.965742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.966551 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:31.966560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:31.967281 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:31.967284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:31.967402 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:31.967433 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:57:31.967959 (+ 526us) client_negotiation.cc:240] Negotiation successful
0927 10:57:31.968039 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":258,"thread_start_us":142,"threads_started":1}
I20260927 10:57:31.968366 16726 raft_consensus.cc:2454] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 0.
I20260927 10:57:31.968886 16994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b2e592c5ce4c9587b6577d98bd41bb" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" is_pre_election: true
I20260927 10:57:31.968896 16793 leader_election.cc:302] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092d377138004a2aa121f50b81bc3c68, 6f36746b42d64e579a5404c69686c137; no voters:
I20260927 10:57:31.969166 16994 raft_consensus.cc:2454] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 0.
I20260927 10:57:31.969203 17064 raft_consensus.cc:2781] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:31.969291 17064 raft_consensus.cc:482] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:31.969345 17064 raft_consensus.cc:3037] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:31.970374 17064 raft_consensus.cc:504] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:31.970744 17064 leader_election.cc:288] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 election: Requested vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575)
I20260927 10:57:31.971040 16994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b2e592c5ce4c9587b6577d98bd41bb" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4ef2fb65d44ea19ffc86248f824f55"
I20260927 10:57:31.971076 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07b2e592c5ce4c9587b6577d98bd41bb" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f36746b42d64e579a5404c69686c137"
I20260927 10:57:31.971171 16994 raft_consensus.cc:3037] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:31.971180 16726 raft_consensus.cc:3037] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:31.972383 16726 raft_consensus.cc:2454] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 1.
I20260927 10:57:31.972383 16994 raft_consensus.cc:2454] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 1.
I20260927 10:57:31.972685 16793 leader_election.cc:302] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092d377138004a2aa121f50b81bc3c68, 6f36746b42d64e579a5404c69686c137; no voters:
I20260927 10:57:31.972878 17064 raft_consensus.cc:2781] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:31.973102 17064 raft_consensus.cc:686] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 1 LEADER]: Becoming Leader. State: Replica: 092d377138004a2aa121f50b81bc3c68, State: Running, Role: LEADER
I20260927 10:57:31.973423 17064 consensus_queue.cc:237] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
W20260927 10:57:31.973758 17041 tablet.cc:2369] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:31.976565 16580 catalog_manager.cc:5495] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 092d377138004a2aa121f50b81bc3c68 (127.1.46.194). New cstate: current_term: 1 leader_uuid: "092d377138004a2aa121f50b81bc3c68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } health_report { overall_health: UNKNOWN } } }
I20260927 10:57:32.009861 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:32.006396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.1.46.194:40381)
0927 10:57:32.006547 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:32.006551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:32.006581 (+ 30us) server_negotiation.cc:407] Connection header received
0927 10:57:32.006814 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:32.006817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:32.006886 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:32.006990 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:32.007241 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:32.007767 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:32.008584 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:32.008789 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:32.008860 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:32.009092 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:32.009478 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:32.009576 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:32.009645 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 10:57:32.009689 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 10:57:32.012858 16907 tablet.cc:2369] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:32.015336 16994 raft_consensus.cc:1261] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Refusing update from remote peer 092d377138004a2aa121f50b81bc3c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:32.015338 16726 raft_consensus.cc:1261] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Refusing update from remote peer 092d377138004a2aa121f50b81bc3c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:32.015964 17064 consensus_queue.cc:1035] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:32.016183 17067 consensus_queue.cc:1035] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:32.026446 17072 mvcc.cc:205] Tried to move back new op lower bound from 7333915246648315904 to 7333915246485962752. Current Snapshot: MvccSnapshot[applied={T|T < 7333915246648315904 or (T in {7333915246648315904})}]
I20260927 10:57:32.031430 16580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:32.031555 16580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:32.033826 16580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Sending DeleteTablet for 3 replicas of tablet 07b2e592c5ce4c9587b6577d98bd41bb
I20260927 10:57:32.034505 16974 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b2e592c5ce4c9587b6577d98bd41bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:32 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:38966
I20260927 10:57:32.034799 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 16643
I20260927 10:57:32.034760 16706 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b2e592c5ce4c9587b6577d98bd41bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:32 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:53326
I20260927 10:57:32.035036 17079 tablet_replica.cc:330] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: stopping tablet replica
I20260927 10:57:32.035471 17079 raft_consensus.cc:2229] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:32.035786 17079 raft_consensus.cc:2258] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:32.037218 16840 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b2e592c5ce4c9587b6577d98bd41bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:32 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:47566
I20260927 10:57:32.037441 17079 ts_tablet_manager.cc:1905] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:32.037684 17080 tablet_replica.cc:330] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: stopping tablet replica
I20260927 10:57:32.037962 17080 raft_consensus.cc:2229] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:57:32.038331 17080 raft_consensus.cc:2258] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:32.039764 17080 ts_tablet_manager.cc:1905] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:32.040827 17079 ts_tablet_manager.cc:1918] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:32.040977 17079 log.cc:1196] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/07b2e592c5ce4c9587b6577d98bd41bb
I20260927 10:57:32.041340 17079 ts_tablet_manager.cc:1939] T 07b2e592c5ce4c9587b6577d98bd41bb P eb4ef2fb65d44ea19ffc86248f824f55: Deleting consensus metadata
I20260927 10:57:32.042560 17080 ts_tablet_manager.cc:1918] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:32.042552 16566 catalog_manager.cc:4841] TS eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621): tablet 07b2e592c5ce4c9587b6577d98bd41bb (table test-table [id=0f9a8d42e2e24d77b7f8f06312fb693a]) successfully deleted
I20260927 10:57:32.042750 17080 log.cc:1196] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/07b2e592c5ce4c9587b6577d98bd41bb
I20260927 10:57:32.043056 17080 ts_tablet_manager.cc:1939] T 07b2e592c5ce4c9587b6577d98bd41bb P 092d377138004a2aa121f50b81bc3c68: Deleting consensus metadata
W20260927 10:57:32.043110 16565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:39575 (error 108)
I20260927 10:57:32.043951 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 16777
I20260927 10:57:32.044061 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:32.043767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39575 (local address 127.0.0.1:53328)
0927 10:57:32.043905 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:32.043980 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 10:57:32.044314 16565 catalog_manager.cc:4584] TS 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07b2e592c5ce4c9587b6577d98bd41bb: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39575: connect: Connection refused (error 111)
I20260927 10:57:32.051121 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 16911
I20260927 10:57:32.051466 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:32.051155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:40381 (local address 127.0.0.1:47576)
0927 10:57:32.051329 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:32.051380 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.194:40381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 10:57:32.057796 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 16549
I20260927 10:57:52.064008 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40117
--webserver_interface=127.1.46.254
--webserver_port=38753
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:57:52.171449 17082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:52.171689 17082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:52.171741 17082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:52.175631 17082 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:57:52.175669 17082 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:57:52.175684 17082 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:57:52.175697 17082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:57:52.175714 17082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:52.175731 17082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:52.175746 17082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:57:52.175760 17082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:57:52.180811 17082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=38753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:52.181679 17082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:52.182564 17082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:52.194887 17091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:52.194895 17088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:52.194958 17089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:52.195267 17082 server_base.cc:1034] running on GCE node
I20260927 10:57:52.195847 17082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:52.196923 17082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:52.198113 17082 hybrid_clock.cc:648] HybridClock initialized: now 1790506672198079 us; error 48 us; skew 500 ppm
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:52.201043 17082 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:57:52.201895 17082 webserver.cc:460] Webserver started at http://127.1.46.254:38753/ using document root <none> and password file <none>
I20260927 10:57:52.202448 17082 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:52.202518 17082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:52.206245 17082 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:52.208393 17098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.209718 17082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:57:52.209865 17082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8eff24e0d9cb47ad951a506657c11258"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "2bd6ae78c1403df906398b4932fb4877"
server_key_iv: "5774a9e83573189e0a9c4a6ed9db0e39"
server_key_version: "encryptionkey@0"
I20260927 10:57:52.210498 17082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:52.222090 17082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:52.225433 17082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:52.225648 17082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:52.232949 17082 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:40117
I20260927 10:57:52.233253 17150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:40117 every 8 connection(s)
I20260927 10:57:52.234053 17082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:57:52.238209 17151 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:57:52.239799 17151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Bootstrap starting.
I20260927 10:57:52.240163 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17082
I20260927 10:57:52.240643 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:39575
--local_ip_for_outbound_sockets=127.1.46.193
--tserver_master_addrs=127.1.46.254:40117
--webserver_port=39021
--webserver_interface=127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:57:52.262617 17151 log.cc:824] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:52.273383 17151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:57:52.273844 17151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Bootstrap complete.
I20260927 10:57:52.279309 17151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:52.279804 17151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eff24e0d9cb47ad951a506657c11258, State: Initialized, Role: FOLLOWER
I20260927 10:57:52.280339 17151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:52.280478 17151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:57:52.280546 17151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:57:52.280650 17151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:57:52.283133 17151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:52.283464 17151 leader_election.cc:302] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eff24e0d9cb47ad951a506657c11258; no voters:
I20260927 10:57:52.283779 17151 leader_election.cc:288] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:57:52.283918 17155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:57:52.284179 17155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 LEADER]: Becoming Leader. State: Replica: 8eff24e0d9cb47ad951a506657c11258, State: Running, Role: LEADER
I20260927 10:57:52.284536 17155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:57:52.284970 17151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:57:52.285934 17156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8eff24e0d9cb47ad951a506657c11258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } } }
I20260927 10:57:52.286062 17156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:52.286684 17164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:57:52.290777 17157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eff24e0d9cb47ad951a506657c11258. Latest consensus state: current_term: 2 leader_uuid: "8eff24e0d9cb47ad951a506657c11258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } } }
I20260927 10:57:52.290915 17157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:57:52.291694 17164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:57:52.292424 17164 catalog_manager.cc:1259] Loaded cluster ID: 0ba30133a5c54afbb915c755a21befd8
I20260927 10:57:52.292481 17164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:57:52.294907 17164 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:57:52.295626 17164 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Loaded TSK: 0
I20260927 10:57:52.309190 17164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Generated new TSK 1
I20260927 10:57:52.309417 17164 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:57:52.369120 17153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:52.369336 17153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:52.369377 17153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:52.373399 17153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:52.373447 17153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:52.373528 17153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:57:52.378655 17153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:39575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=39021
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:52.379557 17153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:52.380472 17153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:52.389672 17179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:52.389783 17178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:52.389871 17153 server_base.cc:1034] running on GCE node
W20260927 10:57:52.389686 17181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:52.390493 17153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:52.391148 17153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:52.392321 17153 hybrid_clock.cc:648] HybridClock initialized: now 1790506672392297 us; error 47 us; skew 500 ppm
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:52.395030 17153 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:57:52.395929 17153 webserver.cc:460] Webserver started at http://127.1.46.193:39021/ using document root <none> and password file <none>
I20260927 10:57:52.396446 17153 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:52.396548 17153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:52.400385 17153 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:57:52.402458 17188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.403755 17153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:52.403843 17153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "6f36746b42d64e579a5404c69686c137"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "6ed167f7cd68cd2e4e182d2c39a5f6e5"
server_key_iv: "f79829757dee3117eb0cf7d4dfa38da2"
server_key_version: "encryptionkey@0"
I20260927 10:57:52.404383 17153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:52.438426 17153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:52.441215 17153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:52.441416 17153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:52.441987 17153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:52.443018 17195 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:57:52.446971 17153 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:57:52.447044 17153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 10:57:52.447125 17153 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:57:52.448765 17153 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:57:52.448830 17153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:57:52.449088 17195 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Bootstrap starting.
I20260927 10:57:52.459151 17153 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:39575
I20260927 10:57:52.459614 17302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:39575 every 8 connection(s)
I20260927 10:57:52.460147 17153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:57:52.467783 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17153
I20260927 10:57:52.467945 17195 log.cc:824] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:52.468286 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:40381
--local_ip_for_outbound_sockets=127.1.46.194
--tserver_master_addrs=127.1.46.254:40117
--webserver_port=38779
--webserver_interface=127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:52.475386 17195 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:57:52.475823 17195 tablet_bootstrap.cc:492] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Bootstrap complete.
I20260927 10:57:52.477025 17195 ts_tablet_manager.cc:1397] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260927 10:57:52.478480 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.462275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:39191 (local address 127.1.46.254:40117)
0927 10:57:52.462683 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.462702 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.463115 (+ 413us) server_negotiation.cc:407] Connection header received
0927 10:57:52.464358 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.464368 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.464632 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.464870 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:52.466360 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.467358 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.468942 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.469401 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.469610 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.472590 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:52.472599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:52.472617 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:52.472654 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:52.474905 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.475548 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.475555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.475564 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.475733 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.476120 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.476125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.476130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.476627 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:52.476780 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.477400 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.477685 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":124,"threads_started":1}
I20260927 10:57:52.479708 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.462237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:39191)
0927 10:57:52.462895 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:52.462973 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:52.463990 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:52.465008 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:52.465017 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:52.465460 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:52.466195 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.466214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.467490 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.467495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.468807 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.468826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.469589 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.469593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.469783 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.470524 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:52.470548 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:52.472346 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:52.475053 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.475059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.475069 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.475408 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.475860 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.475864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.475867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.476013 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.476755 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:52.476770 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:52.477054 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 10:57:52.478726 (+ 1672us) client_negotiation.cc:240] Negotiation successful
0927 10:57:52.478990 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":142,"threads_started":1}
I20260927 10:57:52.479602 17195 raft_consensus.cc:348] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.480295 17195 raft_consensus.cc:729] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f36746b42d64e579a5404c69686c137, State: Initialized, Role: FOLLOWER
I20260927 10:57:52.480760 17195 consensus_queue.cc:260] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.481596 17195 ts_tablet_manager.cc:1428] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:52.482100 17303 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40117
I20260927 10:57:52.482304 17303 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:52.482762 17303 heartbeater.cc:511] Master 127.1.46.254:40117 requested a full tablet report, sending...
I20260927 10:57:52.485074 17115 ts_manager.cc:194] Registered new tserver with Master: 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575)
I20260927 10:57:52.488854 17115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:39191
I20260927 10:57:52.491070 17303 heartbeater.cc:503] Master 127.1.46.254:40117 was elected leader, sending a full tablet report...
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.193@KRBTEST.COM
I20260927 10:57:52.498315 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.488132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39575 (local address 127.0.0.1:42136)
0927 10:57:52.488766 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:52.488796 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:52.488943 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:52.489398 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:52.489403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:52.489768 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:52.489912 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.489919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.491288 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.491291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.492245 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.492257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.492902 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.492906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.493039 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.493511 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:52.493525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:52.494878 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:52.496647 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.496657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.496663 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.496882 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.497186 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.497191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.497193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.497294 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.497726 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:52.497732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:52.497848 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 10:57:52.497979 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 10:57:52.498154 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":117,"threads_started":1}
I20260927 10:57:52.498358 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.488134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.1.46.193:39575)
0927 10:57:52.488538 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.488548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.488880 (+ 332us) server_negotiation.cc:407] Connection header received
0927 10:57:52.489124 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.489127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.489265 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.489397 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:52.490030 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.491142 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.492474 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.492776 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.492855 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.495003 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:52.495014 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:52.495028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:52.495046 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:52.496532 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.496993 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.497007 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.497008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.497095 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.497381 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.497386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.497388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.497614 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:52.497750 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.498064 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.498216 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":119,"threads_started":1}
I20260927 10:57:52.499277 17237 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b2e592c5ce4c9587b6577d98bd41bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:32 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:42136
I20260927 10:57:52.500012 17312 tablet_replica.cc:330] stopping tablet replica
I20260927 10:57:52.500291 17312 raft_consensus.cc:2229] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:52.500509 17312 raft_consensus.cc:2258] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:52.501502 17312 ts_tablet_manager.cc:1905] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:52.505831 17312 ts_tablet_manager.cc:1918] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:52.505934 17312 log.cc:1196] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal/wals/07b2e592c5ce4c9587b6577d98bd41bb
I20260927 10:57:52.506223 17312 ts_tablet_manager.cc:1939] T 07b2e592c5ce4c9587b6577d98bd41bb P 6f36746b42d64e579a5404c69686c137: Deleting consensus metadata
I20260927 10:57:52.508287 17099 catalog_manager.cc:4841] TS 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575): tablet 07b2e592c5ce4c9587b6577d98bd41bb (table test-table [id=0f9a8d42e2e24d77b7f8f06312fb693a]) successfully deleted
W20260927 10:57:52.584256 17308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:52.584496 17308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:52.584661 17308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:52.588779 17308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:52.588827 17308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:52.588883 17308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:57:52.593922 17308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:40381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=38779
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:52.594866 17308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:52.595767 17308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:52.605263 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:52.605275 17317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:52.605273 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:52.605494 17308 server_base.cc:1034] running on GCE node
I20260927 10:57:52.605935 17308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:52.606551 17308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:52.607779 17308 hybrid_clock.cc:648] HybridClock initialized: now 1790506672607728 us; error 60 us; skew 500 ppm
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:52.610541 17308 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:57:52.611438 17308 webserver.cc:460] Webserver started at http://127.1.46.194:38779/ using document root <none> and password file <none>
I20260927 10:57:52.611967 17308 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:52.612068 17308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:52.615724 17308 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:57:52.617561 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.618769 17308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:52.618885 17308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "092d377138004a2aa121f50b81bc3c68"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "dae24c501c74614bd90d14df1816e98a"
server_key_iv: "f68e06636caec9cffd0c32f5253d8d78"
server_key_version: "encryptionkey@0"
I20260927 10:57:52.619395 17308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:52.627667 17308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:52.630805 17308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:52.630990 17308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:52.631554 17308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:52.632308 17308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:52.632387 17308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.632454 17308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:52.632496 17308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.641291 17308 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:40381
I20260927 10:57:52.641336 17440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:40381 every 8 connection(s)
I20260927 10:57:52.642220 17308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:57:52.644994 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17308
I20260927 10:57:52.645473 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:39621
--local_ip_for_outbound_sockets=127.1.46.195
--tserver_master_addrs=127.1.46.254:40117
--webserver_port=39343
--webserver_interface=127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:52.657366 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.643887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:36237 (local address 127.1.46.254:40117)
0927 10:57:52.644023 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.644027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.644943 (+ 916us) server_negotiation.cc:407] Connection header received
0927 10:57:52.645946 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.645953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.646024 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.646123 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:52.647505 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.648089 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.649862 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.650112 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.650192 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.653235 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:52.653242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:52.653246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:52.653280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:52.655086 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.655655 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.655661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.655665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.655767 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.656112 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.656118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.656122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.656392 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:52.656511 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.656973 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.657120 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:57:52.657917 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.644167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:36237)
0927 10:57:52.644789 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:52.644828 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:52.645725 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:52.646271 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:52.646277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:52.646626 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:52.647324 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.647345 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.648222 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.648229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.649728 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.649742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.650283 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.650291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.650542 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.651205 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:52.651231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:52.653054 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:52.655225 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.655229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.655239 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.655533 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.655877 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.655880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.655882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.656010 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.656525 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:52.656530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:52.656736 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 10:57:52.656887 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:57:52.657123 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":108,"threads_started":1}
I20260927 10:57:52.659121 17441 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40117
I20260927 10:57:52.659358 17441 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:52.659852 17441 heartbeater.cc:511] Master 127.1.46.254:40117 requested a full tablet report, sending...
I20260927 10:57:52.660954 17115 ts_manager.cc:194] Registered new tserver with Master: 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381)
I20260927 10:57:52.661670 17115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:36237
W20260927 10:57:52.756949 17444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:57:52.757200 17444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:57:52.757242 17444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:57:52.761483 17444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:57:52.761539 17444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:57:52.761596 17444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:57:52.766554 17444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:39621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=39343
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:57:52.767470 17444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:57:52.768409 17444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:57:52.777498 17452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:52.777508 17449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:57:52.777696 17450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:57:52.777720 17444 server_base.cc:1034] running on GCE node
I20260927 10:57:52.778270 17444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:57:52.778879 17444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:57:52.780072 17444 hybrid_clock.cc:648] HybridClock initialized: now 1790506672780043 us; error 31 us; skew 500 ppm
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:57:52.782780 17444 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:57:52.783591 17444 webserver.cc:460] Webserver started at http://127.1.46.195:39343/ using document root <none> and password file <none>
I20260927 10:57:52.784116 17444 fs_manager.cc:359] Metadata directory not provided
I20260927 10:57:52.784217 17444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:57:52.788189 17444 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:57:52.790133 17459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.791348 17444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:57:52.791474 17444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "eb4ef2fb65d44ea19ffc86248f824f55"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "cc9c9711f7cde58925b0315850e9f1f9"
server_key_iv: "f4b3ea48a031e0e8d0a8ce98253bd273"
server_key_version: "encryptionkey@0"
I20260927 10:57:52.792052 17444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:57:52.803225 17444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:57:52.805768 17444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:57:52.805946 17444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:57:52.806453 17444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:57:52.807258 17444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:57:52.807325 17444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.807389 17444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:57:52.807457 17444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:57:52.816864 17444 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39621
I20260927 10:57:52.816890 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39621 every 8 connection(s)
I20260927 10:57:52.817782 17444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:57:52.822239 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17444
Sep 27 10:57:52 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506672, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:57:52.831418 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.822905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.1.46.254:40117)
0927 10:57:52.823213 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.823218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.823270 (+ 52us) server_negotiation.cc:407] Connection header received
0927 10:57:52.823381 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.823386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.823441 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.823525 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:52.824425 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.825230 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.826258 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.826610 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.826736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.828011 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:52.828015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:52.828019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:52.828050 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:52.829766 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.830181 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.830187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.830190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.830277 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.830555 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.830558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.830561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.830808 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:52.831017 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.831129 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.831249 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260927 10:57:52.832937 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.819773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:34599 (local address 127.1.46.254:40117)
0927 10:57:52.819900 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.819910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.820923 (+ 1013us) server_negotiation.cc:407] Connection header received
0927 10:57:52.821781 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.821785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.821865 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.821973 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:52.823427 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.824290 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.825787 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.826115 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.826229 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.829257 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:52.829260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:52.829262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:52.829294 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:52.830725 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.831296 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.831301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.831305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.831392 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.831784 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.831787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.831789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.831995 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:52.832086 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.832695 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.832804 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:57:52.833788 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.820149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:34599)
0927 10:57:52.820782 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:52.820819 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:52.821575 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:52.822160 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:52.822166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:52.822629 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:57:52.823300 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.823319 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.824435 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.824439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.825676 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.825685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.826294 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.826298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.826483 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.827168 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:57:52.827183 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:57:52.829057 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:57:52.830828 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.830837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.830853 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.831173 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.831493 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:57:52.831497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:57:52.831500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:57:52.831677 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:57:52.832118 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:57:52.832124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:57:52.832414 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 10:57:52.832616 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 10:57:52.832843 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":101,"threads_started":1}
I20260927 10:57:52.834872 17573 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40117
I20260927 10:57:52.835109 17573 heartbeater.cc:464] Registering TS with master...
I20260927 10:57:52.835549 17573 heartbeater.cc:511] Master 127.1.46.254:40117 requested a full tablet report, sending...
I20260927 10:57:52.836699 17115 ts_manager.cc:194] Registered new tserver with Master: eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621)
I20260927 10:57:52.837325 17115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:34599
I20260927 10:57:52.845088 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:57:52.849206 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.845712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.1.46.254:40117)
0927 10:57:52.845853 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.845856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.846014 (+ 158us) server_negotiation.cc:407] Connection header received
0927 10:57:52.846200 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.846204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.846300 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.846405 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:52.846699 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.847205 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.848053 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.848295 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.848401 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.848719 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:52.848831 (+ 112us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 10:57:52.849082 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49048: token expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 10:57:52.849279 17305 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49048: token expired
W20260927 10:57:52.849551 17044 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.46.254:40117)
I20260927 10:57:52.856753 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.849894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.1.46.254:40117)
0927 10:57:52.849996 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.850002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.850054 (+ 52us) server_negotiation.cc:407] Connection header received
0927 10:57:52.850163 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.850168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.850209 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.850291 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:57:52.850929 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.851399 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.852194 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.852403 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.852503 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.853784 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:57:52.853788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:57:52.853793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:57:52.853818 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:57:52.855148 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.855585 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.855589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.855592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.855671 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:57:52.855937 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:57:52.855942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:57:52.855944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:57:52.856117 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:57:52.856250 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.856491 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.856597 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:57:52.859159 17115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:57:52.862286 17115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:57:52.869379 17237 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0cf9f2f1e948f7900845b337aca68e (DEFAULT_TABLE table=test-table [id=4c36a7941e9d4bd39dcda6ea0ed60eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:52.870062 17237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0cf9f2f1e948f7900845b337aca68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:52.872884 17195 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Bootstrap starting.
I20260927 10:57:52.873768 17195 tablet_bootstrap.cc:654] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:52.873894 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.868739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:40381 (local address 127.0.0.1:60772)
0927 10:57:52.868997 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:52.869010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:52.869101 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:52.870093 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:52.870096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:52.870112 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:52.870201 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.870206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.871379 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.871382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.872353 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.872363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.873416 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.873420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.873500 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.873546 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 10:57:52.873638 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 10:57:52.873732 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":54,"thread_start_us":129,"threads_started":1}
I20260927 10:57:52.874178 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.869393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.1.46.194:40381)
0927 10:57:52.869713 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.869725 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.869751 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:57:52.869811 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.869815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.869949 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.870201 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:52.870371 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.871259 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.872475 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.873290 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.873404 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.873544 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.873668 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.873767 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":124,"threads_started":1}
I20260927 10:57:52.874645 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.868516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39621 (local address 127.0.0.1:39354)
0927 10:57:52.868635 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:52.868651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:52.868720 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:52.869712 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:52.869715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:52.869731 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:52.869838 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.869844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.871372 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.871376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.872685 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:52.872695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.874205 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.874210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:52.874338 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.874359 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:57:52.874415 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 10:57:52.874490 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":114}
I20260927 10:57:52.875337 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:52.868741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.1.46.195:39621)
0927 10:57:52.869198 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:52.869210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:52.869232 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:57:52.869302 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:52.869307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:52.869527 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:52.869990 (+ 463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:52.870159 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.871238 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.872809 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:52.873592 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:52.874902 (+ 1310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:52.875055 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:52.875107 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 10:57:52.875202 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":153,"threads_started":1}
I20260927 10:57:52.875397 17195 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: No bootstrap required, opened a new log
I20260927 10:57:52.875469 17195 ts_tablet_manager.cc:1397] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:57:52.875814 17195 raft_consensus.cc:348] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.875903 17195 raft_consensus.cc:374] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:52.875929 17195 raft_consensus.cc:729] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f36746b42d64e579a5404c69686c137, State: Initialized, Role: FOLLOWER
I20260927 10:57:52.876070 17195 consensus_queue.cc:260] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.876322 17195 ts_tablet_manager.cc:1428] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:57:52.877410 17375 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0cf9f2f1e948f7900845b337aca68e (DEFAULT_TABLE table=test-table [id=4c36a7941e9d4bd39dcda6ea0ed60eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:52.878509 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0cf9f2f1e948f7900845b337aca68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:52.884944 17586 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Bootstrap starting.
I20260927 10:57:52.884807 17507 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0cf9f2f1e948f7900845b337aca68e (DEFAULT_TABLE table=test-table [id=4c36a7941e9d4bd39dcda6ea0ed60eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:57:52.885833 17507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0cf9f2f1e948f7900845b337aca68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:57:52.887403 17586 tablet_bootstrap.cc:654] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:52.888098 17586 log.cc:824] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:52.889780 17586 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: No bootstrap required, opened a new log
I20260927 10:57:52.889977 17586 ts_tablet_manager.cc:1397] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:57:52.892843 17588 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Bootstrap starting.
I20260927 10:57:52.893163 17586 raft_consensus.cc:348] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.893440 17586 raft_consensus.cc:374] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:52.893532 17586 raft_consensus.cc:729] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092d377138004a2aa121f50b81bc3c68, State: Initialized, Role: FOLLOWER
I20260927 10:57:52.893978 17586 consensus_queue.cc:260] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.894618 17441 heartbeater.cc:503] Master 127.1.46.254:40117 was elected leader, sending a full tablet report...
I20260927 10:57:52.894925 17586 ts_tablet_manager.cc:1428] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:57:52.895623 17588 tablet_bootstrap.cc:654] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Neither blocks nor log segments found. Creating new log.
I20260927 10:57:52.896499 17588 log.cc:824] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Log is configured to *not* fsync() on all Append() calls
I20260927 10:57:52.898345 17588 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: No bootstrap required, opened a new log
I20260927 10:57:52.898530 17588 ts_tablet_manager.cc:1397] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:57:52.901329 17588 raft_consensus.cc:348] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.901530 17588 raft_consensus.cc:374] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:57:52.901571 17588 raft_consensus.cc:729] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4ef2fb65d44ea19ffc86248f824f55, State: Initialized, Role: FOLLOWER
I20260927 10:57:52.901924 17588 consensus_queue.cc:260] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:52.902504 17573 heartbeater.cc:503] Master 127.1.46.254:40117 was elected leader, sending a full tablet report...
I20260927 10:57:52.902814 17588 ts_tablet_manager.cc:1428] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
W20260927 10:57:52.962399 17304 tablet.cc:2369] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:53.051909 17589 raft_consensus.cc:482] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:57:53.052124 17589 raft_consensus.cc:504] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:53.053253 17589 leader_election.cc:288] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575)
I20260927 10:57:53.057482 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:53.053644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:36151 (local address 127.1.46.195:39621)
0927 10:57:53.053787 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:53.053793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:53.053812 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:57:53.053949 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:53.053955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:53.054011 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:53.054106 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:53.054391 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.055115 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.056320 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.056948 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.057073 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:53.057123 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:53.057304 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:57:53.057388 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:57:53.057509 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:53.053520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39621 (local address 127.1.46.194:36151)
0927 10:57:53.053675 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:53.053692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:53.053798 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:53.054142 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:53.054148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:53.054168 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:53.054291 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:53.054297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.055228 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.055232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:53.056175 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:53.056184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.057061 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.057065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:53.057168 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:53.057193 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:53.057269 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:57:53.057358 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:57:53.058142 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:53.053520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39575 (local address 127.1.46.194:37245)
0927 10:57:53.053984 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:53.053995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:53.054113 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:53.054541 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:57:53.054545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:57:53.054561 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:57:53.054700 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:53.054707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.055727 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.055729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:53.056707 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:57:53.056718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.057777 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.057781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:57:53.057886 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:53.057911 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:57:53.057981 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:57:53.058039 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":134,"thread_start_us":123,"threads_started":1}
I20260927 10:57:53.058210 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:53.053637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:37245 (local address 127.1.46.193:39575)
0927 10:57:53.053930 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:53.053934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:53.054084 (+ 150us) server_negotiation.cc:407] Connection header received
0927 10:57:53.054246 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:53.054251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:53.054317 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:53.054421 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:57:53.054828 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.055559 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.056834 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.057494 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.057612 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:53.057781 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:53.058007 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 10:57:53.058085 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":113,"threads_started":1}
I20260927 10:57:53.058269 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cf9f2f1e948f7900845b337aca68e" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" is_pre_election: true
I20260927 10:57:53.058650 17527 raft_consensus.cc:2454] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 0.
I20260927 10:57:53.058794 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cf9f2f1e948f7900845b337aca68e" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f36746b42d64e579a5404c69686c137" is_pre_election: true
I20260927 10:57:53.059089 17257 raft_consensus.cc:2454] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 0.
I20260927 10:57:53.059159 17329 leader_election.cc:302] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092d377138004a2aa121f50b81bc3c68, eb4ef2fb65d44ea19ffc86248f824f55; no voters:
I20260927 10:57:53.059432 17589 raft_consensus.cc:2781] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:57:53.059554 17589 raft_consensus.cc:482] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:57:53.059617 17589 raft_consensus.cc:3037] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:53.060607 17589 raft_consensus.cc:504] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:53.060931 17589 leader_election.cc:288] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 election: Requested vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575)
I20260927 10:57:53.061247 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cf9f2f1e948f7900845b337aca68e" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f36746b42d64e579a5404c69686c137"
I20260927 10:57:53.061371 17257 raft_consensus.cc:3037] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:53.062316 17257 raft_consensus.cc:2454] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 1.
I20260927 10:57:53.062613 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0cf9f2f1e948f7900845b337aca68e" candidate_uuid: "092d377138004a2aa121f50b81bc3c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4ef2fb65d44ea19ffc86248f824f55"
I20260927 10:57:53.062623 17328 leader_election.cc:302] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092d377138004a2aa121f50b81bc3c68, 6f36746b42d64e579a5404c69686c137; no voters:
I20260927 10:57:53.062744 17527 raft_consensus.cc:3037] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:57:53.062822 17589 raft_consensus.cc:2781] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:57:53.063037 17589 raft_consensus.cc:686] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 1 LEADER]: Becoming Leader. State: Replica: 092d377138004a2aa121f50b81bc3c68, State: Running, Role: LEADER
I20260927 10:57:53.063380 17589 consensus_queue.cc:237] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:57:53.063990 17527 raft_consensus.cc:2454] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092d377138004a2aa121f50b81bc3c68 in term 1.
I20260927 10:57:53.066290 17114 catalog_manager.cc:5495] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 092d377138004a2aa121f50b81bc3c68 (127.1.46.194). New cstate: current_term: 1 leader_uuid: "092d377138004a2aa121f50b81bc3c68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } health_report { overall_health: UNKNOWN } } }
W20260927 10:57:53.068655 17574 tablet.cc:2369] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:57:53.108532 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:57:53.104961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.1.46.194:40381)
0927 10:57:53.105127 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:57:53.105133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:57:53.105149 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:57:53.105231 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:57:53.105233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:57:53.105286 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:57:53.105396 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:57:53.105633 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.106129 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.107034 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:57:53.107269 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:57:53.107376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:57:53.107710 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:57:53.108129 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:57:53.108275 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:57:53.108337 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 10:57:53.108389 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:57:53.114126 17527 raft_consensus.cc:1261] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Refusing update from remote peer 092d377138004a2aa121f50b81bc3c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:53.114328 17257 raft_consensus.cc:1261] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Refusing update from remote peer 092d377138004a2aa121f50b81bc3c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:57:53.114842 17589 consensus_queue.cc:1035] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:53.115022 17594 consensus_queue.cc:1035] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:57:53.120931 17597 mvcc.cc:205] Tried to move back new op lower bound from 7333915333068251136 to 7333915332870692864. Current Snapshot: MvccSnapshot[applied={T|T < 7333915333068251136}]
I20260927 10:57:53.124814 17598 mvcc.cc:205] Tried to move back new op lower bound from 7333915333068251136 to 7333915332870692864. Current Snapshot: MvccSnapshot[applied={T|T < 7333915333068251136}]
I20260927 10:57:53.129662 17115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:53.129825 17115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:57:53.132166 17115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Sending DeleteTablet for 3 replicas of tablet 2e0cf9f2f1e948f7900845b337aca68e
I20260927 10:57:53.132767 17507 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0cf9f2f1e948f7900845b337aca68e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:53 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:39354
I20260927 10:57:53.132913 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0cf9f2f1e948f7900845b337aca68e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:53 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:60772
I20260927 10:57:53.133070 17237 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0cf9f2f1e948f7900845b337aca68e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:57:53 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:42136
I20260927 10:57:53.133291 17606 tablet_replica.cc:330] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: stopping tablet replica
I20260927 10:57:53.133286 17607 tablet_replica.cc:330] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: stopping tablet replica
I20260927 10:57:53.133510 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 17153
I20260927 10:57:53.133558 17607 raft_consensus.cc:2229] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:57:53.133558 17606 raft_consensus.cc:2229] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:57:53.133823 17606 raft_consensus.cc:2258] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:53.133844 17607 raft_consensus.cc:2258] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:57:53.135708 17606 ts_tablet_manager.cc:1905] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:53.137818 17606 ts_tablet_manager.cc:1918] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:53.137926 17606 log.cc:1196] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/2e0cf9f2f1e948f7900845b337aca68e
I20260927 10:57:53.138227 17606 ts_tablet_manager.cc:1939] T 2e0cf9f2f1e948f7900845b337aca68e P eb4ef2fb65d44ea19ffc86248f824f55: Deleting consensus metadata
I20260927 10:57:53.139408 17100 catalog_manager.cc:4841] TS eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621): tablet 2e0cf9f2f1e948f7900845b337aca68e (table test-table [id=4c36a7941e9d4bd39dcda6ea0ed60eca]) successfully deleted
I20260927 10:57:53.139900 17607 ts_tablet_manager.cc:1905] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:57:53.142021 17607 ts_tablet_manager.cc:1918] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:57:53.142156 17607 log.cc:1196] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/2e0cf9f2f1e948f7900845b337aca68e
I20260927 10:57:53.142493 17607 ts_tablet_manager.cc:1939] T 2e0cf9f2f1e948f7900845b337aca68e P 092d377138004a2aa121f50b81bc3c68: Deleting consensus metadata
W20260927 10:57:53.142541 17099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:39575 (error 108)
I20260927 10:57:53.143633 17100 catalog_manager.cc:4841] TS 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381): tablet 2e0cf9f2f1e948f7900845b337aca68e (table test-table [id=4c36a7941e9d4bd39dcda6ea0ed60eca]) successfully deleted
W20260927 10:57:53.143977 17311 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:57:53.143447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39575 (local address 127.0.0.1:42152)
0927 10:57:53.143609 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 10:57:53.143625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:57:53.143704 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:57:53.143906 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39575: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 10:57:53.144173 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 17308
W20260927 10:57:53.144168 17099 catalog_manager.cc:4584] TS 6f36746b42d64e579a5404c69686c137 (127.1.46.193:39575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e0cf9f2f1e948f7900845b337aca68e: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39575: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:57:53.150329 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 17444
I20260927 10:57:53.155578 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 17082
I20260927 10:58:13.162701 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40117
--webserver_interface=127.1.46.254
--webserver_port=38753
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:13.269050 17610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:13.269299 17610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:13.269336 17610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:13.273177 17610 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:58:13.273221 17610 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:58:13.273234 17610 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:58:13.273247 17610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:13.273264 17610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:13.273326 17610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:13.273352 17610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:13.273384 17610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:13.278285 17610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=38753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:13.279193 17610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:13.280164 17610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:13.288512 17618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:13.288552 17616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:13.288512 17615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:13.288594 17610 server_base.cc:1034] running on GCE node
I20260927 10:58:13.289359 17610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:13.290422 17610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:13.291620 17610 hybrid_clock.cc:648] HybridClock initialized: now 1790506693291588 us; error 24 us; skew 500 ppm
Sep 27 10:58:13 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506693, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:13.294265 17610 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:58:13.295166 17610 webserver.cc:460] Webserver started at http://127.1.46.254:38753/ using document root <none> and password file <none>
I20260927 10:58:13.295689 17610 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:13.295773 17610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:13.299333 17610 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:13.301291 17625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.302436 17610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:13.302563 17610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8eff24e0d9cb47ad951a506657c11258"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "2bd6ae78c1403df906398b4932fb4877"
server_key_iv: "5774a9e83573189e0a9c4a6ed9db0e39"
server_key_version: "encryptionkey@0"
I20260927 10:58:13.303146 17610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:13.358366 17610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:13.361541 17610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:13.361748 17610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:13.368477 17610 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:40117
I20260927 10:58:13.368763 17677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:40117 every 8 connection(s)
I20260927 10:58:13.369585 17610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:13.373723 17678 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:58:13.375489 17678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Bootstrap starting.
I20260927 10:58:13.379650 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17610
I20260927 10:58:13.380208 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:39575
--local_ip_for_outbound_sockets=127.1.46.193
--tserver_master_addrs=127.1.46.254:40117
--webserver_port=39021
--webserver_interface=127.1.46.193
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:58:13.394249 17678 log.cc:824] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:13.405869 17678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:58:13.406242 17678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Bootstrap complete.
I20260927 10:58:13.410439 17678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:58:13.410892 17678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eff24e0d9cb47ad951a506657c11258, State: Initialized, Role: FOLLOWER
I20260927 10:58:13.411396 17678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:58:13.411511 17678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:13.411595 17678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:13.411701 17678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:58:13.413424 17678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:58:13.413736 17678 leader_election.cc:302] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eff24e0d9cb47ad951a506657c11258; no voters:
I20260927 10:58:13.414041 17678 leader_election.cc:288] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:58:13.414115 17682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:58:13.414371 17682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [term 3 LEADER]: Becoming Leader. State: Replica: 8eff24e0d9cb47ad951a506657c11258, State: Running, Role: LEADER
I20260927 10:58:13.414714 17682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } }
I20260927 10:58:13.415184 17678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:13.415644 17682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eff24e0d9cb47ad951a506657c11258. Latest consensus state: current_term: 3 leader_uuid: "8eff24e0d9cb47ad951a506657c11258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } } }
I20260927 10:58:13.415759 17682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:13.416121 17683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8eff24e0d9cb47ad951a506657c11258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eff24e0d9cb47ad951a506657c11258" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40117 } } }
I20260927 10:58:13.416285 17683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:13.416226 17687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:13.421548 17687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:13.422232 17687 catalog_manager.cc:1259] Loaded cluster ID: 0ba30133a5c54afbb915c755a21befd8
I20260927 10:58:13.422288 17687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:13.424980 17687 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:13.425650 17687 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Loaded TSK: 0
I20260927 10:58:13.425726 17687 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8eff24e0d9cb47ad951a506657c11258: Loaded TSK: 1
F20260927 10:58:13.426422 17687 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0658eb49ad google::LogMessage::Fail() at ??:0
@ 0x7f0658eb788a google::LogMessage::SendToLog() at ??:0
@ 0x7f0658eb44de google::LogMessage::Flush() at ??:0
@ 0x7f0658eb5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f065bb7818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f065bb70f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f06638197bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f06637f399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f06637f4178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f0663835df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f066385482b std::function<>::operator()() at ??:0
@ 0x7f06637f3b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f06637f4f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f06637f1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0663835640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f065b1063d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f065b106c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f065b1083f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f065b0f7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f065c73b6db start_thread at ??:0
@ 0x7f06579b261f clone at ??:0
W20260927 10:58:13.525738 17680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:13.525993 17680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:13.526078 17680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:13.531308 17680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:13.531404 17680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:13.531522 17680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:58:13.536942 17680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:39575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=39021
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:13.537988 17680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:13.539101 17680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:13.549754 17725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:13.549780 17728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:13.549916 17680 server_base.cc:1034] running on GCE node
W20260927 10:58:13.550113 17726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:13.550681 17680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:13.551337 17680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:13.552516 17680 hybrid_clock.cc:648] HybridClock initialized: now 1790506693552491 us; error 42 us; skew 500 ppm
Sep 27 10:58:13 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506693, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:13.555343 17680 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:58:13.556347 17680 webserver.cc:460] Webserver started at http://127.1.46.193:39021/ using document root <none> and password file <none>
I20260927 10:58:13.557067 17680 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:13.557160 17680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:13.562026 17680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:58:13.564487 17735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.565955 17680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:58:13.566073 17680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "6f36746b42d64e579a5404c69686c137"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "6ed167f7cd68cd2e4e182d2c39a5f6e5"
server_key_iv: "f79829757dee3117eb0cf7d4dfa38da2"
server_key_version: "encryptionkey@0"
I20260927 10:58:13.566802 17680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:13.585909 17680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:13.589521 17680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:13.589723 17680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:13.590474 17680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:13.591781 17742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:58:13.595832 17680 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:58:13.595906 17680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 10:58:13.595964 17680 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:58:13.598027 17680 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:58:13.598096 17680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:58:13.598385 17742 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Bootstrap starting.
I20260927 10:58:13.608791 17680 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:39575
I20260927 10:58:13.609138 17849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:39575 every 8 connection(s)
I20260927 10:58:13.610108 17680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:58:13.616999 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17680
I20260927 10:58:13.617498 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:40381
--local_ip_for_outbound_sockets=127.1.46.194
--tserver_master_addrs=127.1.46.254:40117
--webserver_port=38779
--webserver_interface=127.1.46.194
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:58:13.617981 17742 log.cc:824] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:13.624939 17742 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:58:13.625286 17742 tablet_bootstrap.cc:492] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Bootstrap complete.
I20260927 10:58:13.626358 17742 ts_tablet_manager.cc:1397] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.009s
I20260927 10:58:13.628713 17742 raft_consensus.cc:348] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:58:13.629104 17742 raft_consensus.cc:729] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f36746b42d64e579a5404c69686c137, State: Initialized, Role: FOLLOWER
I20260927 10:58:13.629563 17742 consensus_queue.cc:260] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:58:13.630445 17742 ts_tablet_manager.cc:1428] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 10:58:13.662935 17852 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:58:13.612368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:37351)
0927 10:58:13.612986 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.613066 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:13.614128 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:13.661930 (+ 47802us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":139,"threads_started":1}
W20260927 10:58:13.663667 17850 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:40117 (0 consecutive failures): Network error: Failed to ping master at 127.1.46.254:40117: Client connection negotiation failed: client connection to 127.1.46.254:40117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:58:13.664505 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.664221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:45399)
0927 10:58:13.664364 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.664440 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:58:13.665287 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.665032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:55933)
0927 10:58:13.665159 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.665220 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 10:58:13.665526 17850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:58:13.740685 17854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:13.740901 17854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:13.740939 17854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:13.744796 17854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:13.744879 17854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:13.744966 17854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:58:13.749771 17854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:40381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=38779
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:13.750658 17854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:13.751632 17854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:13.760594 17861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:13.760594 17860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:13.760653 17863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:13.760919 17854 server_base.cc:1034] running on GCE node
I20260927 10:58:13.761353 17854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:13.761929 17854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:13.763098 17854 hybrid_clock.cc:648] HybridClock initialized: now 1790506693763083 us; error 26 us; skew 500 ppm
Sep 27 10:58:13 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506693, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:13.765935 17854 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:58:13.766824 17854 webserver.cc:460] Webserver started at http://127.1.46.194:38779/ using document root <none> and password file <none>
I20260927 10:58:13.767357 17854 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:13.767416 17854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:13.771130 17854 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:58:13.773193 17870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.774348 17854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:13.774464 17854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "092d377138004a2aa121f50b81bc3c68"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "dae24c501c74614bd90d14df1816e98a"
server_key_iv: "f68e06636caec9cffd0c32f5253d8d78"
server_key_version: "encryptionkey@0"
I20260927 10:58:13.775007 17854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:13.789363 17854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:13.792084 17854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:13.792271 17854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:13.792822 17854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:13.793609 17854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:13.793654 17854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.793722 17854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:13.793762 17854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.802407 17854 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:40381
I20260927 10:58:13.802443 17983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:40381 every 8 connection(s)
I20260927 10:58:13.803323 17854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:58:13.806519 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.805196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:41943)
0927 10:58:13.805798 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.805894 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":96,"threads_started":1}
W20260927 10:58:13.806943 17984 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:40117 (0 consecutive failures): Network error: Failed to ping master at 127.1.46.254:40117: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
I20260927 10:58:13.807423 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.807243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:40327)
0927 10:58:13.807333 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.807375 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 10:58:13.807936 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.807742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:52469)
0927 10:58:13.807845 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.807882 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 10:58:13.808120 17984 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:58:13.809265 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17854
I20260927 10:58:13.809772 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:39621
--local_ip_for_outbound_sockets=127.1.46.195
--tserver_master_addrs=127.1.46.254:40117
--webserver_port=39343
--webserver_interface=127.1.46.195
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:13.914822 17987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:13.915040 17987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:13.915077 17987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:13.919029 17987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:13.919077 17987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:13.919132 17987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:58:13.924026 17987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:39621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=39343
--tserver_master_addrs=127.1.46.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:13.924921 17987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:13.925837 17987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:13.934813 17992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:13.934798 17993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:13.934847 17995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:13.935073 17987 server_base.cc:1034] running on GCE node
I20260927 10:58:13.935542 17987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:13.936126 17987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:13.937302 17987 hybrid_clock.cc:648] HybridClock initialized: now 1790506693937285 us; error 28 us; skew 500 ppm
Sep 27 10:58:13 dist-test-slave-cffh krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506693, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:13.939988 17987 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:58:13.940917 17987 webserver.cc:460] Webserver started at http://127.1.46.195:39343/ using document root <none> and password file <none>
I20260927 10:58:13.941457 17987 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:13.941543 17987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:13.945356 17987 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:13.947372 18002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.948684 17987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:13.948839 17987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "eb4ef2fb65d44ea19ffc86248f824f55"
format_stamp: "Formatted at 2026-09-27 10:57:31 on dist-test-slave-cffh"
server_key: "cc9c9711f7cde58925b0315850e9f1f9"
server_key_iv: "f4b3ea48a031e0e8d0a8ce98253bd273"
server_key_version: "encryptionkey@0"
I20260927 10:58:13.949385 17987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:13.972955 17987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:13.976116 17987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:13.976317 17987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:13.976897 17987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:13.977771 17987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:13.977849 17987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.977924 17987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:13.977981 17987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:13.987936 17987 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:39621
I20260927 10:58:13.987957 18115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:39621 every 8 connection(s)
I20260927 10:58:13.988909 17987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:58:13.992331 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.990965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:59701)
0927 10:58:13.991581 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.991685 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":108,"threads_started":1}
W20260927 10:58:13.992806 18116 heartbeater.cc:644] Failed to heartbeat to 127.1.46.254:40117 (0 consecutive failures): Network error: Failed to ping master at 127.1.46.254:40117: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
I20260927 10:58:13.993323 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.993131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:58609)
0927 10:58:13.993230 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.993273 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 10:58:13.994019 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:13.993845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:43169)
0927 10:58:13.993938 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:13.993977 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:58:13.994185 18116 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:58:13.995873 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 17987
I20260927 10:58:14.666479 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:14.666087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:47135)
0927 10:58:14.666322 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:14.666373 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260927 10:58:14.809020 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:14.808653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:60591)
0927 10:58:14.808882 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:14.808932 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260927 10:58:14.995079 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:14.994694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:54655)
0927 10:58:14.994910 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:14.994965 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
I20260927 10:58:15.304663 18123 raft_consensus.cc:482] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:15.304853 18123 raft_consensus.cc:504] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:58:15.305909 18123 leader_election.cc:288] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381)
Sep 27 10:58:15 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506693, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.194@KRBTEST.COM
Sep 27 10:58:15 dist-test-slave-cffh krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506693, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.195@KRBTEST.COM
I20260927 10:58:15.325850 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:15.306146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:39621 (local address 127.1.46.193:34919)
0927 10:58:15.306800 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:15.306815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:15.306931 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:15.308437 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:15.308443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:15.308961 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:15.309387 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:15.309396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.310712 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.310715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:15.311608 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:15.311617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.312284 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.312287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:15.312513 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:15.319107 (+ 6594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:15.319124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:15.321218 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:15.323439 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:15.323441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:15.323442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:15.323698 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:15.324033 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:15.324038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:15.324041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:15.324193 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:15.324881 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:15.324899 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:15.325092 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 10:58:15.325286 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:58:15.325652 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":57,"spinlock_wait_cycles":25344,"thread_start_us":69,"threads_started":1}
I20260927 10:58:15.325865 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:15.306676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:37311 (local address 127.1.46.194:40381)
0927 10:58:15.307207 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:15.307228 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:15.307258 (+ 30us) server_negotiation.cc:407] Connection header received
0927 10:58:15.307852 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:15.307857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:15.308141 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:15.308433 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:15.309561 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.310495 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.311716 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.312172 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.312340 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:15.321058 (+ 8718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:15.321065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:15.321082 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:15.321128 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:15.323144 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:15.323857 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:15.323861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:15.323865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:15.323965 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:15.324294 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:15.324298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:15.324301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:15.324699 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:15.324836 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:15.325400 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 10:58:15.325655 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":80,"threads_started":1}
I20260927 10:58:15.325898 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:15.306277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:34919 (local address 127.1.46.195:39621)
0927 10:58:15.306700 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:15.306732 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:15.306929 (+ 197us) server_negotiation.cc:407] Connection header received
0927 10:58:15.307720 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:15.307726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:15.308074 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:15.308362 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:15.309566 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.310503 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.311748 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.312167 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.312324 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:15.321435 (+ 9111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:15.321439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:15.321450 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:15.321476 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:15.323321 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:15.323808 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:15.323814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:15.323821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:15.323936 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:15.324289 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:15.324293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:15.324298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:15.324719 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:15.324902 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:15.325399 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 10:58:15.325703 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":53,"threads_started":1}
I20260927 10:58:15.325901 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:15.306515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:40381 (local address 127.1.46.193:37311)
0927 10:58:15.306924 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:15.306938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:15.307056 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:15.308438 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:15.308443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:15.308951 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:15.309380 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:15.309393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.310675 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.310682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:15.311600 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:15.311609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:15.312310 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:15.312314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:15.312518 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:15.319155 (+ 6637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:15.319168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:15.320789 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:15.323261 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:15.323266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:15.323280 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:15.323709 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:15.324094 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:15.324096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:15.324097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:15.324192 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:15.324826 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:15.324832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:15.325093 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 10:58:15.325286 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 10:58:15.325733 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":396,"spinlock_wait_cycles":23808,"thread_start_us":82,"threads_started":1}
W20260927 10:58:15.328758 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
W20260927 10:58:15.328969 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
I20260927 10:58:15.329032 17737 leader_election.cc:302] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f36746b42d64e579a5404c69686c137; no voters: 092d377138004a2aa121f50b81bc3c68, eb4ef2fb65d44ea19ffc86248f824f55
I20260927 10:58:15.329290 18123 raft_consensus.cc:2726] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:58:15.667641 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:15.667332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:36977)
0927 10:58:15.667486 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:15.667551 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 10:58:15.810032 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:15.809661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:41083)
0927 10:58:15.809902 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:15.809947 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260927 10:58:15.996094 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:15.995722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:36809)
0927 10:58:15.995962 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:15.996011 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260927 10:58:16.668783 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:16.668369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:39719)
0927 10:58:16.668633 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:16.668689 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260927 10:58:16.811311 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:16.810889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:43731)
0927 10:58:16.811144 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:16.811204 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
I20260927 10:58:16.971028 18132 raft_consensus.cc:482] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:16.971150 18132 raft_consensus.cc:504] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:58:16.971498 18132 leader_election.cc:288] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381)
W20260927 10:58:16.972378 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
W20260927 10:58:16.972533 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
I20260927 10:58:16.972605 17737 leader_election.cc:302] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f36746b42d64e579a5404c69686c137; no voters: 092d377138004a2aa121f50b81bc3c68, eb4ef2fb65d44ea19ffc86248f824f55
I20260927 10:58:16.972747 18132 raft_consensus.cc:2726] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:58:16.997253 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:16.996851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:40207)
0927 10:58:16.997097 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:16.997154 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260927 10:58:17.669981 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:17.669585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:53475)
0927 10:58:17.669840 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:17.669890 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260927 10:58:17.812382 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:17.811983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:53953)
0927 10:58:17.812233 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:17.812285 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260927 10:58:17.998420 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:17.998027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:47975)
0927 10:58:17.998282 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:17.998335 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260927 10:58:18.552940 18137 raft_consensus.cc:482] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:18.553105 18137 raft_consensus.cc:504] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:58:18.553646 18137 leader_election.cc:288] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381)
W20260927 10:58:18.554365 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
W20260927 10:58:18.554507 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
I20260927 10:58:18.554612 17737 leader_election.cc:302] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f36746b42d64e579a5404c69686c137; no voters: 092d377138004a2aa121f50b81bc3c68, eb4ef2fb65d44ea19ffc86248f824f55
I20260927 10:58:18.554760 18137 raft_consensus.cc:2726] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:58:18.671399 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:18.670923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:56109)
0927 10:58:18.671251 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:18.671304 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":147,"threads_started":1}
I20260927 10:58:18.813580 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:18.813087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:48591)
0927 10:58:18.813390 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:18.813461 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":65,"threads_started":1}
I20260927 10:58:18.999531 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:18.999156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:36789)
0927 10:58:18.999400 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:18.999450 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260927 10:58:19.672745 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:19.672321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:45727)
0927 10:58:19.672591 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:19.672646 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
I20260927 10:58:19.814815 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:19.814356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:38749)
0927 10:58:19.814651 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:19.814725 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260927 10:58:20.000578 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:20.000222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:57813)
0927 10:58:20.000451 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:20.000499 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260927 10:58:20.673873 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:20.673458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:37615)
0927 10:58:20.673712 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:20.673773 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260927 10:58:20.816005 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:20.815527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:54233)
0927 10:58:20.815855 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:20.815912 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":94,"threads_started":1}
I20260927 10:58:21.001834 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:21.001352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:44659)
0927 10:58:21.001680 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:21.001734 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":83,"threads_started":1}
I20260927 10:58:21.675171 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:21.674751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:37455)
0927 10:58:21.675022 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:21.675074 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260927 10:58:21.817186 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:21.816774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:54595)
0927 10:58:21.817031 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:21.817086 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260927 10:58:22.002996 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:22.002617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:55065)
0927 10:58:22.002856 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:22.002906 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260927 10:58:22.676321 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:22.675913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:49573)
0927 10:58:22.676175 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:22.676229 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260927 10:58:22.696336 18151 raft_consensus.cc:482] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:22.696456 18151 raft_consensus.cc:504] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb4ef2fb65d44ea19ffc86248f824f55" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 39621 } } peers { permanent_uuid: "092d377138004a2aa121f50b81bc3c68" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 40381 } } peers { permanent_uuid: "6f36746b42d64e579a5404c69686c137" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39575 } }
I20260927 10:58:22.696957 18151 leader_election.cc:288] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621), 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381)
W20260927 10:58:22.697718 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 092d377138004a2aa121f50b81bc3c68 (127.1.46.194:40381): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
W20260927 10:58:22.697847 17737 leader_election.cc:341] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb4ef2fb65d44ea19ffc86248f824f55 (127.1.46.195:39621): Not found: Tablet not found: 2e0cf9f2f1e948f7900845b337aca68e
I20260927 10:58:22.697916 17737 leader_election.cc:302] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f36746b42d64e579a5404c69686c137; no voters: 092d377138004a2aa121f50b81bc3c68, eb4ef2fb65d44ea19ffc86248f824f55
I20260927 10:58:22.698103 18151 raft_consensus.cc:2726] T 2e0cf9f2f1e948f7900845b337aca68e P 6f36746b42d64e579a5404c69686c137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:58:22.818403 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:22.817974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:53079)
0927 10:58:22.818262 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:22.818315 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
I20260927 10:58:23.004261 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:23.003767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:44351)
0927 10:58:23.004066 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:23.004136 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260927 10:58:23.677587 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:23.677150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:57383)
0927 10:58:23.677418 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:23.677491 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260927 10:58:23.819667 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:23.819167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:34413)
0927 10:58:23.819492 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:23.819551 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":121,"threads_started":1}
I20260927 10:58:24.005383 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:24.004990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:49337)
0927 10:58:24.005240 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:24.005294 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260927 10:58:24.678762 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:24.678305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:57961)
0927 10:58:24.678538 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:24.678668 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260927 10:58:24.820870 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:24.820410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:57219)
0927 10:58:24.820705 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:24.820764 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":73,"threads_started":1}
I20260927 10:58:25.006495 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:25.006088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:49707)
0927 10:58:25.006344 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:25.006404 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260927 10:58:25.679955 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:25.679543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:50435)
0927 10:58:25.679799 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:25.679855 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260927 10:58:25.821959 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:25.821560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:49059)
0927 10:58:25.821818 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:25.821868 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
I20260927 10:58:26.007582 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:26.007208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:52095)
0927 10:58:26.007444 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:26.007494 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260927 10:58:26.681083 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:26.680683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:59721)
0927 10:58:26.680941 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:26.680992 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260927 10:58:26.823045 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:26.822675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:51493)
0927 10:58:26.822902 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:26.822955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260927 10:58:27.008697 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:27.008284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:37139)
0927 10:58:27.008561 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:27.008610 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260927 10:58:27.682163 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:27.681762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:57039)
0927 10:58:27.681997 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:27.682051 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260927 10:58:27.824131 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:27.823744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:48303)
0927 10:58:27.823992 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:27.824042 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260927 10:58:28.009763 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:28.009381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:49309)
0927 10:58:28.009623 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:28.009674 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260927 10:58:28.683585 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:28.683186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.193:49901)
0927 10:58:28.683444 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:28.683494 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260927 10:58:28.825307 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:28.824912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.194:51143)
0927 10:58:28.825160 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:28.825215 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260927 10:58:29.003480 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 17680
I20260927 10:58:29.009181 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 17854
I20260927 10:58:29.011236 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:29.010880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40117 (local address 127.1.46.195:49707)
0927 10:58:29.011097 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:29.011156 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.254:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260927 10:58:29.014716 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 17987
2026-09-27T10:58:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59274 ms)
[----------] 30 tests from SecurityITest (236257 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:29 dist-test-slave-cffh krb5kdc[18174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:29 dist-test-slave-cffh krb5kdc[18174](info): set up 2 sockets
Sep 27 10:58:29 dist-test-slave-cffh krb5kdc[18174](info): commencing operation
krb5kdc: starting...
W20260927 10:58:30.058384 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:58:30.177651 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43765
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:43765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:30.283402 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:30.283661 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:30.283735 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:30.287582 18187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:30.287654 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:30.287690 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:30.287752 18187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:30.287792 18187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:30.292649 18187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:43765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:30.293622 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:30.294561 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:30.302500 18192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:30.302524 18193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:30.302690 18187 server_base.cc:1034] running on GCE node
W20260927 10:58:30.302527 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:30.303369 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:30.304422 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:30.305603 18187 hybrid_clock.cc:648] HybridClock initialized: now 1790506710305581 us; error 35 us; skew 500 ppm
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:30.308166 18187 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:58:30.309032 18187 webserver.cc:460] Webserver started at http://127.1.46.254:42441/ using document root <none> and password file <none>
I20260927 10:58:30.309551 18187 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:30.309638 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:30.309870 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:30.311857 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "f26f1d900cdf438ca856388053ec1861"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "0e96cdab638a9f981eb24f2fae43f8ea"
server_key_iv: "09257973d58115deedc4e0a6d1ad6d7c"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.312395 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "f26f1d900cdf438ca856388053ec1861"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "0e96cdab638a9f981eb24f2fae43f8ea"
server_key_iv: "09257973d58115deedc4e0a6d1ad6d7c"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.315884 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:58:30.318063 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.319209 18187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:30.319339 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "f26f1d900cdf438ca856388053ec1861"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "0e96cdab638a9f981eb24f2fae43f8ea"
server_key_iv: "09257973d58115deedc4e0a6d1ad6d7c"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.319442 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:30.329156 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:30.331640 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:30.331835 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:30.338420 18187 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:43765
I20260927 10:58:30.338799 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:43765 every 8 connection(s)
I20260927 10:58:30.339354 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:30.341804 18255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:30.343137 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 18187
I20260927 10:58:30.343288 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:30.343560 1211 external_mini_cluster.cc:1428] Setting key 24bce78149a0b5b2349865058469d2c0
I20260927 10:58:30.347703 18255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861: Bootstrap starting.
I20260927 10:58:30.350069 18255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:30.350952 18255 log.cc:824] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:30.352928 18255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861: No bootstrap required, opened a new log
I20260927 10:58:30.358440 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:30.344776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.1.46.254:43765)
0927 10:58:30.345194 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:30.345210 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:30.345248 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:58:30.346026 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:30.346046 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:30.346411 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:30.346751 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:30.347344 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.348618 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.349434 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.350104 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.350275 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:30.352780 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:30.352784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:30.352794 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:30.352816 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:30.354592 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:30.355157 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:30.355163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:30.355170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:30.355338 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:30.356002 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:30.356005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:30.356006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:30.356458 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:30.356614 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:30.357067 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 10:58:30.357391 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
I20260927 10:58:30.358538 18255 raft_consensus.cc:348] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26f1d900cdf438ca856388053ec1861" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43765 } }
I20260927 10:58:30.358865 18255 raft_consensus.cc:374] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:30.358942 18255 raft_consensus.cc:729] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26f1d900cdf438ca856388053ec1861, State: Initialized, Role: FOLLOWER
I20260927 10:58:30.359614 18255 consensus_queue.cc:260] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26f1d900cdf438ca856388053ec1861" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43765 } }
I20260927 10:58:30.359774 18255 raft_consensus.cc:388] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:30.359921 18255 raft_consensus.cc:482] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:30.360037 18255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:30.361261 18255 raft_consensus.cc:504] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26f1d900cdf438ca856388053ec1861" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43765 } }
I20260927 10:58:30.361611 18255 leader_election.cc:302] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f26f1d900cdf438ca856388053ec1861; no voters:
I20260927 10:58:30.362152 18255 leader_election.cc:288] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:30.362592 18261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:30.362895 18261 raft_consensus.cc:686] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [term 1 LEADER]: Becoming Leader. State: Replica: f26f1d900cdf438ca856388053ec1861, State: Running, Role: LEADER
I20260927 10:58:30.363281 18261 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26f1d900cdf438ca856388053ec1861" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43765 } }
I20260927 10:58:30.363682 18255 sys_catalog.cc:564] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:30.365136 18260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f26f1d900cdf438ca856388053ec1861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26f1d900cdf438ca856388053ec1861" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43765 } } }
I20260927 10:58:30.365279 18260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:30.365635 18262 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26f1d900cdf438ca856388053ec1861. Latest consensus state: current_term: 1 leader_uuid: "f26f1d900cdf438ca856388053ec1861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26f1d900cdf438ca856388053ec1861" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 43765 } } }
I20260927 10:58:30.365754 18262 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:30.366780 18269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:30.370560 18269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:30.377208 18269 catalog_manager.cc:1347] Generated new cluster ID: 72dc09ff614544e69d9578ad8c70705d
I20260927 10:58:30.377265 18269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:30.385766 18269 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:30.387375 18269 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:30.396337 18269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861: Generated new TSK 0
I20260927 10:58:30.396965 18269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:58:30.433291 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:43765
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:30.542145 18281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:30.542368 18281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:30.542419 18281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:30.546494 18281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:30.546553 18281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:30.546715 18281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:58:30.551977 18281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:43765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:30.553052 18281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:30.554049 18281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:30.564016 18286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:30.564018 18289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:30.564002 18287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:30.564288 18281 server_base.cc:1034] running on GCE node
I20260927 10:58:30.564777 18281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:30.565429 18281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:30.566672 18281 hybrid_clock.cc:648] HybridClock initialized: now 1790506710566644 us; error 43 us; skew 500 ppm
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:30.569391 18281 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:58:30.570180 18281 webserver.cc:460] Webserver started at http://127.1.46.193:36897/ using document root <none> and password file <none>
I20260927 10:58:30.570822 18281 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:30.570884 18281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:30.571087 18281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:30.573009 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "ccc4709ec28e445a9fbe2a66384e2511"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "e87c1250ffab8c1b255fd7c1f16c3606"
server_key_iv: "83774b9f3dc4e165ddc88586fa99f506"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.573542 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "ccc4709ec28e445a9fbe2a66384e2511"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "e87c1250ffab8c1b255fd7c1f16c3606"
server_key_iv: "83774b9f3dc4e165ddc88586fa99f506"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.577111 18281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 10:58:30.579473 18296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.580649 18281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:58:30.580783 18281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "ccc4709ec28e445a9fbe2a66384e2511"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "e87c1250ffab8c1b255fd7c1f16c3606"
server_key_iv: "83774b9f3dc4e165ddc88586fa99f506"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.580886 18281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:30.605072 18281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:30.608136 18281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:30.608363 18281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:30.608964 18281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:30.609817 18281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:30.609894 18281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.609977 18281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:30.610028 18281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.619261 18281 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:37623
I20260927 10:58:30.619328 18409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:37623 every 8 connection(s)
I20260927 10:58:30.620216 18281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:58:30.629283 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 18281
I20260927 10:58:30.629416 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:58:30.629765 1211 external_mini_cluster.cc:1428] Setting key c256387ad581a6310f75fdebdb461c2c
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:30.635612 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:30.622021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:47199 (local address 127.1.46.254:43765)
0927 10:58:30.622183 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:30.622189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:30.623168 (+ 979us) server_negotiation.cc:407] Connection header received
0927 10:58:30.624222 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:30.624225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:30.624282 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:30.624413 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:30.625663 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.626254 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.627522 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.627777 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.627897 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:30.630652 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:30.630656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:30.630658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:30.630680 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:30.632132 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:30.632731 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:30.632734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:30.632735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:30.632802 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:30.634374 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:30.634380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:30.634385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:30.634666 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:30.634765 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:30.635208 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 10:58:30.635341 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:58:30.636282 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:30.622455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:43765 (local address 127.1.46.193:47199)
0927 10:58:30.623018 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:30.623061 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:30.623994 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:30.624621 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:30.624628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:30.624993 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:30.625500 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:30.625513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.626405 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.626408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:30.627392 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:30.627401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.627931 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.627936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:30.628129 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:30.628694 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:30.628711 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:30.630418 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:30.632245 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:30.632253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:30.632269 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:30.632637 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:30.634153 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:30.634155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:30.634157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:30.634281 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:30.634773 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:30.634778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:30.634982 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 10:58:30.635127 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 10:58:30.635367 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":186,"threads_started":1}
I20260927 10:58:30.637671 18410 heartbeater.cc:347] Connected to a master server at 127.1.46.254:43765
I20260927 10:58:30.637933 18410 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:30.638420 18410 heartbeater.cc:511] Master 127.1.46.254:43765 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:58:30.640149 18219 ts_manager.cc:194] Registered new tserver with Master: ccc4709ec28e445a9fbe2a66384e2511 (127.1.46.193:37623)
I20260927 10:58:30.641363 18219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:47199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:58:30.657259 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:43765
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:30.760679 18415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:30.760908 18415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:30.760996 18415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:30.765086 18415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:30.765159 18415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:30.765259 18415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:58:30.770215 18415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:43765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:30.771132 18415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:30.772010 18415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:30.780751 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:30.780848 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:30.780786 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:30.781068 18415 server_base.cc:1034] running on GCE node
I20260927 10:58:30.781498 18415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:30.782073 18415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:30.783258 18415 hybrid_clock.cc:648] HybridClock initialized: now 1790506710783234 us; error 42 us; skew 500 ppm
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:30.785897 18415 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:58:30.786705 18415 webserver.cc:460] Webserver started at http://127.1.46.194:41757/ using document root <none> and password file <none>
I20260927 10:58:30.787220 18415 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:30.787307 18415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:30.787503 18415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:30.789687 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "4299858b088742d38f2196429672cbe0"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "3fdc4e3a6f433a1205c0061ecf322958"
server_key_iv: "8c4ac819533cbe8b977c51370fbcf705"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.790270 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "4299858b088742d38f2196429672cbe0"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "3fdc4e3a6f433a1205c0061ecf322958"
server_key_iv: "8c4ac819533cbe8b977c51370fbcf705"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.794010 18415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:58:30.796417 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.797675 18415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:30.797851 18415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "4299858b088742d38f2196429672cbe0"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "3fdc4e3a6f433a1205c0061ecf322958"
server_key_iv: "8c4ac819533cbe8b977c51370fbcf705"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.797967 18415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:30.807740 18415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:30.810177 18415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:30.810340 18415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:30.810911 18415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:30.811583 18415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:30.811659 18415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.811715 18415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:30.811764 18415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.820308 18415 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:46243
I20260927 10:58:30.820369 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:46243 every 8 connection(s)
I20260927 10:58:30.821130 18415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:58:30.822377 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 18415
I20260927 10:58:30.822504 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:58:30.822927 1211 external_mini_cluster.cc:1428] Setting key 15f66410456910382fea2c34e5180372
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
I20260927 10:58:30.834362 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:30.822896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:57385 (local address 127.1.46.254:43765)
0927 10:58:30.823030 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:30.823035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:30.824005 (+ 970us) server_negotiation.cc:407] Connection header received
0927 10:58:30.824972 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:30.824975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:30.825044 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:30.825135 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:30.826220 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.826812 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.827839 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.828064 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.828189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:30.830830 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:30.830835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:30.830841 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:30.830873 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:30.832454 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:30.832920 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:30.832927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:30.832932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:30.833038 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:30.833311 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:30.833313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:30.833314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:30.833492 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:30.833610 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:30.834069 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 10:58:30.834202 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
I20260927 10:58:30.835290 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:30.823236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:43765 (local address 127.1.46.194:57385)
0927 10:58:30.823782 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:30.823885 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:30.824780 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:30.825286 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:30.825292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:30.825617 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:30.826076 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:30.826087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.826940 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.826948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:30.827722 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:30.827730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:30.828211 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:30.828215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:30.828394 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:30.828934 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:30.828960 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:30.830673 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:30.832561 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:30.832566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:30.832575 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:30.832798 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:30.833129 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:30.833131 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:30.833132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:30.833241 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:30.833624 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:30.833632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:30.833851 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 10:58:30.834028 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:58:30.834371 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260927 10:58:30.836484 18544 heartbeater.cc:347] Connected to a master server at 127.1.46.254:43765
I20260927 10:58:30.836720 18544 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:30.837289 18544 heartbeater.cc:511] Master 127.1.46.254:43765 requested a full tablet report, sending...
I20260927 10:58:30.838413 18219 ts_manager.cc:194] Registered new tserver with Master: 4299858b088742d38f2196429672cbe0 (127.1.46.194:46243)
I20260927 10:58:30.839278 18219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:57385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:58:30.849766 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:43765
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:30.955411 18549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:30.955623 18549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:30.955662 18549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:30.959451 18549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:30.959492 18549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:30.959544 18549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:58:30.964362 18549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:43765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:30.965245 18549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:30.966109 18549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:30.975004 18557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:30.975026 18554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:30.975005 18555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:30.975245 18549 server_base.cc:1034] running on GCE node
I20260927 10:58:30.975670 18549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:30.976289 18549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:30.977425 18549 hybrid_clock.cc:648] HybridClock initialized: now 1790506710977402 us; error 43 us; skew 500 ppm
Sep 27 10:58:30 dist-test-slave-cffh krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:30.980486 18549 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:58:30.981310 18549 webserver.cc:460] Webserver started at http://127.1.46.195:46107/ using document root <none> and password file <none>
I20260927 10:58:30.981818 18549 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:30.981876 18549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:30.982096 18549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:30.984023 18549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "0af8874789f2407e99e16c2e90e67a87"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "a4e33b651c6a7bfacc912f56d9542bdb"
server_key_iv: "556823170a96286e487593e8df0aa33a"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.984611 18549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "0af8874789f2407e99e16c2e90e67a87"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "a4e33b651c6a7bfacc912f56d9542bdb"
server_key_iv: "556823170a96286e487593e8df0aa33a"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.988112 18549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:30.990319 18564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:30.991614 18549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:30.991772 18549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "0af8874789f2407e99e16c2e90e67a87"
format_stamp: "Formatted at 2026-09-27 10:58:30 on dist-test-slave-cffh"
server_key: "a4e33b651c6a7bfacc912f56d9542bdb"
server_key_iv: "556823170a96286e487593e8df0aa33a"
server_key_version: "encryptionkey@0"
I20260927 10:58:30.991886 18549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:31.005297 18549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:31.007915 18549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:31.008096 18549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:31.008616 18549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:31.009353 18549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:31.009452 18549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:31.009516 18549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:31.009564 18549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:31.018800 18549 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:41981
I20260927 10:58:31.018826 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:41981 every 8 connection(s)
I20260927 10:58:31.019605 18549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:58:31.025264 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 18549
I20260927 10:58:31.025418 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:58:31.025815 1211 external_mini_cluster.cc:1428] Setting key 8ec9114f364051d0e6bb057cf37e01f1
Sep 27 10:58:31 dist-test-slave-cffh krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506710, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:31.033341 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.021274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:40079 (local address 127.1.46.254:43765)
0927 10:58:31.021436 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.021441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.022569 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 10:58:31.023643 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.023646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.023691 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.023762 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:31.024896 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.025467 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.027219 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.027513 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.027622 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.030061 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:31.030064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:31.030066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:31.030089 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:31.031604 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:31.032073 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:31.032075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:31.032076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:31.032143 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:31.032432 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:31.032434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:31.032435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:31.032577 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:31.032689 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.033137 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.033233 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:58:31.034034 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.021640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:43765 (local address 127.1.46.195:40079)
0927 10:58:31.022370 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.022441 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:31.023402 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:31.023919 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:31.023925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:31.024270 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:31.024768 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.024782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.025664 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.025669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.027053 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.027065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.027647 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.027650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.027846 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.028434 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:31.028451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:31.029899 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:31.031702 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:31.031706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:31.031716 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:31.031988 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:31.032224 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:31.032227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:31.032228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:31.032353 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:31.032709 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:31.032713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:31.032909 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 10:58:31.033063 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 10:58:31.033284 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":100,"threads_started":1}
I20260927 10:58:31.035132 18678 heartbeater.cc:347] Connected to a master server at 127.1.46.254:43765
I20260927 10:58:31.035367 18678 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:31.035868 18678 heartbeater.cc:511] Master 127.1.46.254:43765 requested a full tablet report, sending...
I20260927 10:58:31.036855 18219 ts_manager.cc:194] Registered new tserver with Master: 0af8874789f2407e99e16c2e90e67a87 (127.1.46.195:41981)
I20260927 10:58:31.037441 18219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:40079
I20260927 10:58:31.039644 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:58:31.048475 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.041650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.1.46.254:43765)
0927 10:58:31.041750 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.041753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.041981 (+ 228us) server_negotiation.cc:407] Connection header received
0927 10:58:31.042076 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.042079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.042111 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.042173 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:31.042925 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.043440 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.044221 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.044405 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.044474 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.045690 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:31.045693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:31.045695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:31.045713 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:31.047151 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:31.047520 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:31.047522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:31.047523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:31.047585 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:31.047807 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:31.047809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:31.047810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:31.047976 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:31.048068 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.048268 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.048379 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:58:31.051093 18219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:58:31.054009 18219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:58:31.068257 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.061853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:41981 (local address 127.0.0.1:45072)
0927 10:58:31.062192 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.062241 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:31.062374 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:31.063878 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:31.063881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:31.063897 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:31.064004 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.064012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.065509 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.065513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.066770 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.066782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.067732 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.067736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.067872 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.067893 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:58:31.067956 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:58:31.068074 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":9,"thread_start_us":65,"threads_started":1}
I20260927 10:58:31.068264 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.062888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:37623 (local address 127.0.0.1:47836)
0927 10:58:31.063225 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.063236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:31.063341 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:31.064096 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:31.064100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:31.064120 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:31.064209 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.064216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.065657 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.065660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.066852 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.066862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.067660 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.067663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.067816 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.067865 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 10:58:31.068009 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 10:58:31.068108 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":128,"thread_start_us":82,"threads_started":1}
I20260927 10:58:31.068643 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.062031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:46243 (local address 127.0.0.1:39576)
0927 10:58:31.062940 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.062956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:31.063037 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:31.063576 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:31.063584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:31.063609 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:31.063748 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.063756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.065138 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.065144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.066693 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.066702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.067732 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.067736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.068236 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.068254 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:58:31.068338 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:58:31.068562 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":617,"thread_start_us":99,"threads_started":1}
I20260927 10:58:31.068297 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.063055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.1.46.193:37623)
0927 10:58:31.063655 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.063665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.063683 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:58:31.063740 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.063743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.063909 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.064107 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:31.064314 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.065500 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.066953 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.067518 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.067660 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.067831 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.068044 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.068156 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":140,"threads_started":1}
I20260927 10:58:31.069250 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.061996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.1.46.195:41981)
0927 10:58:31.063403 (+ 1407us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.063417 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.063441 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:58:31.063514 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.063522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.063731 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.063895 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:31.064128 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.065372 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.066887 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.067600 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.068841 (+ 1241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.068962 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.069008 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.069095 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":148,"threads_started":1}
I20260927 10:58:31.070117 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.062516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.1.46.194:46243)
0927 10:58:31.063018 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.063032 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.063076 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:58:31.063163 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.063171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.063384 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.063899 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:31.064078 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.064999 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.066801 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.067601 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.067797 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.067925 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.069901 (+ 1976us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.069986 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":164,"threads_started":1}
I20260927 10:58:31.071782 18612 tablet_service.cc:1467] Processing CreateTablet for tablet 08edbbec19cb4d99bd93e26fb5339a61 (DEFAULT_TABLE table=test-table [id=757db79a460f4d7b9dfa17c6679c143b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:31.071784 18344 tablet_service.cc:1467] Processing CreateTablet for tablet 08edbbec19cb4d99bd93e26fb5339a61 (DEFAULT_TABLE table=test-table [id=757db79a460f4d7b9dfa17c6679c143b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:31.072901 18612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08edbbec19cb4d99bd93e26fb5339a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:31.072897 18344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08edbbec19cb4d99bd93e26fb5339a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:31.076234 18478 tablet_service.cc:1467] Processing CreateTablet for tablet 08edbbec19cb4d99bd93e26fb5339a61 (DEFAULT_TABLE table=test-table [id=757db79a460f4d7b9dfa17c6679c143b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:31.077098 18478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08edbbec19cb4d99bd93e26fb5339a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:31.080348 18695 tablet_bootstrap.cc:492] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: Bootstrap starting.
I20260927 10:58:31.081553 18696 tablet_bootstrap.cc:492] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Bootstrap starting.
I20260927 10:58:31.082752 18695 tablet_bootstrap.cc:654] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:31.084018 18696 tablet_bootstrap.cc:654] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:31.084031 18695 log.cc:824] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:31.084688 18697 tablet_bootstrap.cc:492] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Bootstrap starting.
I20260927 10:58:31.084975 18696 log.cc:824] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:31.086112 18695 tablet_bootstrap.cc:492] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: No bootstrap required, opened a new log
I20260927 10:58:31.086304 18695 ts_tablet_manager.cc:1397] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260927 10:58:31.086828 18696 tablet_bootstrap.cc:492] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: No bootstrap required, opened a new log
I20260927 10:58:31.087023 18696 ts_tablet_manager.cc:1397] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:58:31.087695 18697 tablet_bootstrap.cc:654] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:31.088848 18697 log.cc:824] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:31.090051 18696 raft_consensus.cc:348] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.090337 18696 raft_consensus.cc:374] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:31.090408 18696 raft_consensus.cc:729] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af8874789f2407e99e16c2e90e67a87, State: Initialized, Role: FOLLOWER
I20260927 10:58:31.090240 18695 raft_consensus.cc:348] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.090487 18695 raft_consensus.cc:374] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:31.090544 18695 raft_consensus.cc:729] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc4709ec28e445a9fbe2a66384e2511, State: Initialized, Role: FOLLOWER
I20260927 10:58:31.090775 18697 tablet_bootstrap.cc:492] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: No bootstrap required, opened a new log
I20260927 10:58:31.090926 18697 ts_tablet_manager.cc:1397] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:58:31.090950 18696 consensus_queue.cc:260] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.090967 18695 consensus_queue.cc:260] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.091652 18678 heartbeater.cc:503] Master 127.1.46.254:43765 was elected leader, sending a full tablet report...
I20260927 10:58:31.091894 18410 heartbeater.cc:503] Master 127.1.46.254:43765 was elected leader, sending a full tablet report...
I20260927 10:58:31.091952 18695 ts_tablet_manager.cc:1428] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:58:31.092760 18696 ts_tablet_manager.cc:1428] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:58:31.094782 18697 raft_consensus.cc:348] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.095049 18697 raft_consensus.cc:374] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:31.095124 18697 raft_consensus.cc:729] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4299858b088742d38f2196429672cbe0, State: Initialized, Role: FOLLOWER
I20260927 10:58:31.095543 18697 consensus_queue.cc:260] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.096170 18544 heartbeater.cc:503] Master 127.1.46.254:43765 was elected leader, sending a full tablet report...
I20260927 10:58:31.096470 18697 ts_tablet_manager.cc:1428] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:58:31.121610 18411 tablet.cc:2369] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:31.270552 18679 tablet.cc:2369] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:31.322235 18545 tablet.cc:2369] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:31.323462 18703 raft_consensus.cc:482] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:31.323647 18703 raft_consensus.cc:504] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.324743 18703 leader_election.cc:288] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0af8874789f2407e99e16c2e90e67a87 (127.1.46.195:41981), ccc4709ec28e445a9fbe2a66384e2511 (127.1.46.193:37623)
I20260927 10:58:31.329828 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.325072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:36765 (local address 127.1.46.193:37623)
0927 10:58:31.325208 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.325213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.325230 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:58:31.325332 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.325340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.325383 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.325481 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:31.325834 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.326608 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.328465 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.329069 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.329189 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.329234 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.329623 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.329706 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:58:31.329828 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.325175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:48485 (local address 127.1.46.195:41981)
0927 10:58:31.325303 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.325307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.325455 (+ 148us) server_negotiation.cc:407] Connection header received
0927 10:58:31.325602 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.325606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.325655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.325713 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:31.326004 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.326721 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.328465 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.329078 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.329189 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.329234 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.329641 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.329715 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:58:31.329954 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.324966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:37623 (local address 127.1.46.194:36765)
0927 10:58:31.325105 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.325123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:31.325222 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:31.325515 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:31.325520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:31.325540 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:31.325681 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.325687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.326725 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.326731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.328318 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.328331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.329200 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.329204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.329457 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.329508 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 10:58:31.329597 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:58:31.329782 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":396}
I20260927 10:58:31.330195 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.324996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:41981 (local address 127.1.46.194:48485)
0927 10:58:31.325363 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.325374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:31.325487 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:31.325755 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:31.325759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:31.325782 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:31.325904 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.325909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.326814 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.326817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.328318 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.328331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.329200 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.329204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.329450 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.329511 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 10:58:31.329595 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:58:31.329775 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":251,"thread_start_us":85,"threads_started":1}
I20260927 10:58:31.330682 18364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08edbbec19cb4d99bd93e26fb5339a61" candidate_uuid: "4299858b088742d38f2196429672cbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc4709ec28e445a9fbe2a66384e2511" is_pre_election: true
I20260927 10:58:31.330765 18632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08edbbec19cb4d99bd93e26fb5339a61" candidate_uuid: "4299858b088742d38f2196429672cbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af8874789f2407e99e16c2e90e67a87" is_pre_election: true
I20260927 10:58:31.330997 18364 raft_consensus.cc:2454] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4299858b088742d38f2196429672cbe0 in term 0.
I20260927 10:58:31.331041 18632 raft_consensus.cc:2454] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4299858b088742d38f2196429672cbe0 in term 0.
I20260927 10:58:31.331498 18432 leader_election.cc:302] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af8874789f2407e99e16c2e90e67a87, 4299858b088742d38f2196429672cbe0; no voters:
I20260927 10:58:31.331761 18703 raft_consensus.cc:2781] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:58:31.331827 18703 raft_consensus.cc:482] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:58:31.331854 18703 raft_consensus.cc:3037] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:31.332811 18703 raft_consensus.cc:504] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.333138 18703 leader_election.cc:288] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 0af8874789f2407e99e16c2e90e67a87 (127.1.46.195:41981), ccc4709ec28e445a9fbe2a66384e2511 (127.1.46.193:37623)
I20260927 10:58:31.333473 18632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08edbbec19cb4d99bd93e26fb5339a61" candidate_uuid: "4299858b088742d38f2196429672cbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af8874789f2407e99e16c2e90e67a87"
I20260927 10:58:31.333493 18364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08edbbec19cb4d99bd93e26fb5339a61" candidate_uuid: "4299858b088742d38f2196429672cbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc4709ec28e445a9fbe2a66384e2511"
I20260927 10:58:31.333611 18632 raft_consensus.cc:3037] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:31.333611 18364 raft_consensus.cc:3037] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:31.334995 18364 raft_consensus.cc:2454] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4299858b088742d38f2196429672cbe0 in term 1.
I20260927 10:58:31.334995 18632 raft_consensus.cc:2454] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4299858b088742d38f2196429672cbe0 in term 1.
I20260927 10:58:31.335306 18432 leader_election.cc:302] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af8874789f2407e99e16c2e90e67a87, 4299858b088742d38f2196429672cbe0; no voters:
I20260927 10:58:31.335526 18703 raft_consensus.cc:2781] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:31.335741 18703 raft_consensus.cc:686] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 1 LEADER]: Becoming Leader. State: Replica: 4299858b088742d38f2196429672cbe0, State: Running, Role: LEADER
I20260927 10:58:31.336125 18703 consensus_queue.cc:237] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } }
I20260927 10:58:31.338917 18219 catalog_manager.cc:5495] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4299858b088742d38f2196429672cbe0 (127.1.46.194). New cstate: current_term: 1 leader_uuid: "4299858b088742d38f2196429672cbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4299858b088742d38f2196429672cbe0" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 } health_report { overall_health: UNKNOWN } } }
I20260927 10:58:31.356564 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.353081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.1.46.194:46243)
0927 10:58:31.353195 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.353201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.353214 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:58:31.353384 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.353386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.353449 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.353560 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:58:31.353842 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.354353 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.355229 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.355439 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.355541 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.355839 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:58:31.356202 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:58:31.356321 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.356381 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.356428 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:58:31.361781 18632 raft_consensus.cc:1261] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 1 FOLLOWER]: Refusing update from remote peer 4299858b088742d38f2196429672cbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:31.361781 18364 raft_consensus.cc:1261] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 1 FOLLOWER]: Refusing update from remote peer 4299858b088742d38f2196429672cbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:31.362459 18703 consensus_queue.cc:1035] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af8874789f2407e99e16c2e90e67a87" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:31.362691 18705 consensus_queue.cc:1035] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc4709ec28e445a9fbe2a66384e2511" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:31.378435 18219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:31.378615 18219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:31.380996 18219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f26f1d900cdf438ca856388053ec1861: Sending DeleteTablet for 3 replicas of tablet 08edbbec19cb4d99bd93e26fb5339a61
I20260927 10:58:31.381688 18612 tablet_service.cc:1514] Processing DeleteTablet for tablet 08edbbec19cb4d99bd93e26fb5339a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:31 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:45072
I20260927 10:58:31.381944 18478 tablet_service.cc:1514] Processing DeleteTablet for tablet 08edbbec19cb4d99bd93e26fb5339a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:31 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:39576
I20260927 10:58:31.382141 18717 tablet_replica.cc:330] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: stopping tablet replica
I20260927 10:58:31.382463 18718 tablet_replica.cc:330] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: stopping tablet replica
I20260927 10:58:31.382489 18717 raft_consensus.cc:2229] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:31.382480 18344 tablet_service.cc:1514] Processing DeleteTablet for tablet 08edbbec19cb4d99bd93e26fb5339a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:31 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:47836
I20260927 10:58:31.382817 18717 raft_consensus.cc:2258] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:31.383217 18719 tablet_replica.cc:330] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511: stopping tablet replica
I20260927 10:58:31.383474 18719 raft_consensus.cc:2229] T 08edbbec19cb4d99bd93e26fb5339a61 P ccc4709ec28e445a9fbe2a66384e2511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:31.383512 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 18281
I20260927 10:58:31.384218 18717 ts_tablet_manager.cc:1905] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:31.386837 18717 ts_tablet_manager.cc:1918] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:31.386936 18717 log.cc:1196] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/08edbbec19cb4d99bd93e26fb5339a61
I20260927 10:58:31.387326 18717 ts_tablet_manager.cc:1939] T 08edbbec19cb4d99bd93e26fb5339a61 P 0af8874789f2407e99e16c2e90e67a87: Deleting consensus metadata
I20260927 10:58:31.388381 18204 catalog_manager.cc:4841] TS 0af8874789f2407e99e16c2e90e67a87 (127.1.46.195:41981): tablet 08edbbec19cb4d99bd93e26fb5339a61 (table test-table [id=757db79a460f4d7b9dfa17c6679c143b]) successfully deleted
I20260927 10:58:31.382861 18718 raft_consensus.cc:2229] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:58:31.391412 18718 raft_consensus.cc:2258] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:31.392544 18718 ts_tablet_manager.cc:1905] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:58:31.393615 18205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:37623 (error 108)
I20260927 10:58:31.394675 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.394343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:37623 (local address 127.0.0.1:47840)
0927 10:58:31.394472 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.394540 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:58:31.394805 18205 catalog_manager.cc:4584] TS ccc4709ec28e445a9fbe2a66384e2511 (127.1.46.193:37623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08edbbec19cb4d99bd93e26fb5339a61: Network error: Client connection negotiation failed: client connection to 127.1.46.193:37623: connect: Connection refused (error 111)
I20260927 10:58:31.395018 18718 ts_tablet_manager.cc:1918] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:31.395166 18718 log.cc:1196] T 08edbbec19cb4d99bd93e26fb5339a61 P 4299858b088742d38f2196429672cbe0: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/08edbbec19cb4d99bd93e26fb5339a61
I20260927 10:58:31.395196 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 18415
I20260927 10:58:31.400956 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 18549
I20260927 10:58:31.401768 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.401162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:46243 (local address 127.0.0.1:39596)
0927 10:58:31.401659 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.401707 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.194:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399}
I20260927 10:58:31.407061 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 18187
2026-09-27T10:58:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2392 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T10:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:31Z Disabled control of system clock
I20260927 10:58:31.436409 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45227
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:58:31.541620 18724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:31.541836 18724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:31.541867 18724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:31.545595 18724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:31.545640 18724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:31.545660 18724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:31.545673 18724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:31.545687 18724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:31.550354 18724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:31.551282 18724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:31.552434 18724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:31.560428 18730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:31.560580 18724 server_base.cc:1034] running on GCE node
W20260927 10:58:31.560422 18729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:31.560427 18732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:31.561278 18724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:31.562284 18724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:31.563474 18724 hybrid_clock.cc:648] HybridClock initialized: now 1790506711563464 us; error 34 us; skew 500 ppm
I20260927 10:58:31.565222 18724 webserver.cc:460] Webserver started at http://127.1.46.254:37239/ using document root <none> and password file <none>
I20260927 10:58:31.565778 18724 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:31.565871 18724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:31.566108 18724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:31.568089 18724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "520d934d68ed4957baf3c79076b4217a"
format_stamp: "Formatted at 2026-09-27 10:58:31 on dist-test-slave-cffh"
server_key: "30f192a1d4cf7595eab476a41d2aee01"
server_key_iv: "389360b761739e00c8a004f60ccc16f8"
server_key_version: "encryptionkey@0"
I20260927 10:58:31.568643 18724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "520d934d68ed4957baf3c79076b4217a"
format_stamp: "Formatted at 2026-09-27 10:58:31 on dist-test-slave-cffh"
server_key: "30f192a1d4cf7595eab476a41d2aee01"
server_key_iv: "389360b761739e00c8a004f60ccc16f8"
server_key_version: "encryptionkey@0"
I20260927 10:58:31.572028 18724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:31.574254 18738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:31.575426 18724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:31.575557 18724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "520d934d68ed4957baf3c79076b4217a"
format_stamp: "Formatted at 2026-09-27 10:58:31 on dist-test-slave-cffh"
server_key: "30f192a1d4cf7595eab476a41d2aee01"
server_key_iv: "389360b761739e00c8a004f60ccc16f8"
server_key_version: "encryptionkey@0"
I20260927 10:58:31.575677 18724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:31.611194 18724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:31.611953 18724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:31.612175 18724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:31.619506 18724 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45227
I20260927 10:58:31.619525 18790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45227 every 8 connection(s)
I20260927 10:58:31.620441 18724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:31.621891 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 18724
I20260927 10:58:31.622037 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:31.622372 1211 external_mini_cluster.cc:1428] Setting key 1adbb88bfee55fbfc09e5c8e3700c42b
I20260927 10:58:31.623844 18791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:31.629992 18791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a: Bootstrap starting.
I20260927 10:58:31.631966 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.623755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.1.46.254:45227)
0927 10:58:31.624456 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.624474 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.624511 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:58:31.625128 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.625142 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.625507 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.625779 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:31.626126 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.627146 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.628330 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.628764 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.628885 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.629773 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:31.629778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:31.629790 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:31.630245 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:31.630318 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:31.630349 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:31.630457 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.630963 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.631264 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":188,"threads_started":1}
I20260927 10:58:31.632619 18791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:31.633298 18791 log.cc:824] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:31.635138 18791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a: No bootstrap required, opened a new log
I20260927 10:58:31.638201 18791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520d934d68ed4957baf3c79076b4217a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45227 } }
I20260927 10:58:31.638424 18791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:31.638516 18791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520d934d68ed4957baf3c79076b4217a, State: Initialized, Role: FOLLOWER
I20260927 10:58:31.638938 18791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520d934d68ed4957baf3c79076b4217a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45227 } }
I20260927 10:58:31.639078 18791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:31.639164 18791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:31.639278 18791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:31.640090 18791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520d934d68ed4957baf3c79076b4217a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45227 } }
I20260927 10:58:31.640399 18791 leader_election.cc:302] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 520d934d68ed4957baf3c79076b4217a; no voters:
I20260927 10:58:31.640697 18791 leader_election.cc:288] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:31.640801 18796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:31.641045 18796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [term 1 LEADER]: Becoming Leader. State: Replica: 520d934d68ed4957baf3c79076b4217a, State: Running, Role: LEADER
I20260927 10:58:31.641320 18796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520d934d68ed4957baf3c79076b4217a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45227 } }
I20260927 10:58:31.641816 18791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:31.642863 18798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 520d934d68ed4957baf3c79076b4217a. Latest consensus state: current_term: 1 leader_uuid: "520d934d68ed4957baf3c79076b4217a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520d934d68ed4957baf3c79076b4217a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45227 } } }
I20260927 10:58:31.642985 18798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:31.643358 18805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:31.643325 18797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "520d934d68ed4957baf3c79076b4217a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520d934d68ed4957baf3c79076b4217a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45227 } } }
I20260927 10:58:31.643522 18797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:31.646310 18805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:31.651741 18805 catalog_manager.cc:1347] Generated new cluster ID: c7ace478421a4632b8d49b193ee3c01f
I20260927 10:58:31.651816 18805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:31.670054 18805 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:31.671248 18805 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:31.682400 18805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a: Generated new TSK 0
I20260927 10:58:31.683188 18805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:31.695073 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45227
--builtin_ntp_servers=127.1.46.212:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:58:31.821135 18815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:31.821367 18815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:31.821404 18815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:31.825402 18815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:31.825452 18815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:31.825541 18815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:58:31.830392 18815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:31.831382 18815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:31.832662 18815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:31.841863 18820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:31.841840 18823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:31.841879 18821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:31.842007 18815 server_base.cc:1034] running on GCE node
I20260927 10:58:31.842494 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:31.843112 18815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:31.844298 18815 hybrid_clock.cc:648] HybridClock initialized: now 1790506711844292 us; error 34 us; skew 500 ppm
I20260927 10:58:31.846017 18815 webserver.cc:460] Webserver started at http://127.1.46.193:37217/ using document root <none> and password file <none>
I20260927 10:58:31.846546 18815 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:31.846696 18815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:31.846928 18815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:31.848973 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "b891f299908b4a898313d9260b167733"
format_stamp: "Formatted at 2026-09-27 10:58:31 on dist-test-slave-cffh"
server_key: "ecee0bddc0a469b77d33acebde6462d1"
server_key_iv: "4211538bbecbda6e06882329f9be0f85"
server_key_version: "encryptionkey@0"
I20260927 10:58:31.849510 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "b891f299908b4a898313d9260b167733"
format_stamp: "Formatted at 2026-09-27 10:58:31 on dist-test-slave-cffh"
server_key: "ecee0bddc0a469b77d33acebde6462d1"
server_key_iv: "4211538bbecbda6e06882329f9be0f85"
server_key_version: "encryptionkey@0"
I20260927 10:58:31.852813 18815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:31.855118 18829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:31.856282 18815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:31.856418 18815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "b891f299908b4a898313d9260b167733"
format_stamp: "Formatted at 2026-09-27 10:58:31 on dist-test-slave-cffh"
server_key: "ecee0bddc0a469b77d33acebde6462d1"
server_key_iv: "4211538bbecbda6e06882329f9be0f85"
server_key_version: "encryptionkey@0"
I20260927 10:58:31.856524 18815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:31.890141 18815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:31.890875 18815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:31.891083 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:31.891657 18815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:31.892529 18815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:31.892597 18815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:31.892671 18815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:31.892711 18815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:31.901289 18815 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:45607
I20260927 10:58:31.901314 18942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:45607 every 8 connection(s)
I20260927 10:58:31.902175 18815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:58:31.911329 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.903983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:49803 (local address 127.1.46.254:45227)
0927 10:58:31.904118 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:31.904122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:31.905101 (+ 979us) server_negotiation.cc:407] Connection header received
0927 10:58:31.906556 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:31.906559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:31.906662 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:31.906769 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:31.907623 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.908180 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.909347 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.909585 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.909716 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.910852 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:31.910856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:31.910858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:31.910869 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:31.910887 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:31.910894 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:31.910972 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:31.911123 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 10:58:31.911190 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:58:31.912147 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 18815
I20260927 10:58:31.912181 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:31.904364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45227 (local address 127.1.46.193:49803)
0927 10:58:31.904961 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:31.904998 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:31.906344 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:31.906951 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:31.906957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:31.907037 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:31.907487 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.907499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.908322 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.908325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.909219 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:31.909229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:31.909756 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:31.909760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:31.909917 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:31.910526 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:31.910527 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:31.910565 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:31.910701 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:31.910974 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:31.910985 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:31.910993 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:58:31.911123 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 10:58:31.911387 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260927 10:58:31.912276 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:58:31.912567 1211 external_mini_cluster.cc:1428] Setting key c6c421f7ea8e439d571986c1f44e48fb
I20260927 10:58:31.913491 18943 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45227
I20260927 10:58:31.913758 18943 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:31.914330 18943 heartbeater.cc:511] Master 127.1.46.254:45227 requested a full tablet report, sending...
I20260927 10:58:31.914633 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45227
--builtin_ntp_servers=127.1.46.212:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:58:31.915840 18755 ts_manager.cc:194] Registered new tserver with Master: b891f299908b4a898313d9260b167733 (127.1.46.193:45607)
I20260927 10:58:31.916986 18755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.193:49803
W20260927 10:58:32.020691 18946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:32.020920 18946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:32.020956 18946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:32.024902 18946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:32.024945 18946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:32.025033 18946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:58:32.029747 18946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:32.030697 18946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:32.031986 18946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:32.041088 18952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:32.041129 18954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:32.041287 18946 server_base.cc:1034] running on GCE node
W20260927 10:58:32.041129 18951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:32.041828 18946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:32.042341 18946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:32.043488 18946 hybrid_clock.cc:648] HybridClock initialized: now 1790506712043431 us; error 67 us; skew 500 ppm
I20260927 10:58:32.045203 18946 webserver.cc:460] Webserver started at http://127.1.46.194:35663/ using document root <none> and password file <none>
I20260927 10:58:32.045737 18946 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:32.045791 18946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:32.045974 18946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:32.047927 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "8e9b9e0280fe41f193518cf040e65428"
format_stamp: "Formatted at 2026-09-27 10:58:32 on dist-test-slave-cffh"
server_key: "b489a8c4450cb7941844c3fd25b630ff"
server_key_iv: "a0881f98c987f8306fc851217cf4f6ca"
server_key_version: "encryptionkey@0"
I20260927 10:58:32.048571 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "8e9b9e0280fe41f193518cf040e65428"
format_stamp: "Formatted at 2026-09-27 10:58:32 on dist-test-slave-cffh"
server_key: "b489a8c4450cb7941844c3fd25b630ff"
server_key_iv: "a0881f98c987f8306fc851217cf4f6ca"
server_key_version: "encryptionkey@0"
I20260927 10:58:32.052053 18946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:32.054184 18960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:32.055301 18946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:32.055425 18946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "8e9b9e0280fe41f193518cf040e65428"
format_stamp: "Formatted at 2026-09-27 10:58:32 on dist-test-slave-cffh"
server_key: "b489a8c4450cb7941844c3fd25b630ff"
server_key_iv: "a0881f98c987f8306fc851217cf4f6ca"
server_key_version: "encryptionkey@0"
I20260927 10:58:32.055526 18946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:32.074653 18946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:32.075313 18946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:32.075505 18946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:32.076040 18946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:32.076882 18946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:32.076951 18946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:32.077015 18946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:32.077064 18946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:32.085455 18946 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:34021
I20260927 10:58:32.085523 19073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:34021 every 8 connection(s)
I20260927 10:58:32.086341 18946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:58:32.090407 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 18946
I20260927 10:58:32.090564 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:58:32.090893 1211 external_mini_cluster.cc:1428] Setting key 9ea382ee6f269dbe326ee9d70f9c1ad5
I20260927 10:58:32.092806 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45227
--builtin_ntp_servers=127.1.46.212:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:58:32.095804 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.088060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:42365 (local address 127.1.46.254:45227)
0927 10:58:32.088224 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.088229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.089269 (+ 1040us) server_negotiation.cc:407] Connection header received
0927 10:58:32.090703 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.090708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.090773 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.090820 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:32.091892 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.092533 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.093799 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.094023 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.094136 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.095279 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:32.095283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:32.095287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:32.095299 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:32.095326 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:32.095337 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:32.095442 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.095596 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.095668 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:58:32.096593 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.088515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45227 (local address 127.1.46.194:42365)
0927 10:58:32.089132 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.089169 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:32.090430 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:32.091202 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:32.091209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:32.091291 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:32.091762 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.091774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.092701 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.092709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.093672 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.093684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.094175 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.094179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.094334 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.095010 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:32.095012 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:32.095050 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:32.095110 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:32.095444 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:32.095455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:32.095463 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:58:32.095596 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 10:58:32.095859 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260927 10:58:32.097865 19074 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45227
I20260927 10:58:32.098079 19074 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:32.098557 19074 heartbeater.cc:511] Master 127.1.46.254:45227 requested a full tablet report, sending...
I20260927 10:58:32.099635 18755 ts_manager.cc:194] Registered new tserver with Master: 8e9b9e0280fe41f193518cf040e65428 (127.1.46.194:34021)
I20260927 10:58:32.100188 18755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.194:42365
W20260927 10:58:32.201591 19077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:32.201834 19077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:32.201898 19077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:32.205739 19077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:32.205809 19077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:32.205904 19077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:58:32.210347 19077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:32.211251 19077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:32.212425 19077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:32.221335 19082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:32.221341 19083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:32.221376 19077 server_base.cc:1034] running on GCE node
W20260927 10:58:32.221344 19085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:32.221942 19077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:32.222517 19077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:32.223730 19077 hybrid_clock.cc:648] HybridClock initialized: now 1790506712223703 us; error 41 us; skew 500 ppm
I20260927 10:58:32.225397 19077 webserver.cc:460] Webserver started at http://127.1.46.195:41431/ using document root <none> and password file <none>
I20260927 10:58:32.225971 19077 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:32.226058 19077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:32.226289 19077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:32.228348 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "b35c8c88e67b40ed9f6c011013db2952"
format_stamp: "Formatted at 2026-09-27 10:58:32 on dist-test-slave-cffh"
server_key: "012358e305830cc942f684c6197c3492"
server_key_iv: "dfab030e988821d41292908092f9b790"
server_key_version: "encryptionkey@0"
I20260927 10:58:32.228912 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "b35c8c88e67b40ed9f6c011013db2952"
format_stamp: "Formatted at 2026-09-27 10:58:32 on dist-test-slave-cffh"
server_key: "012358e305830cc942f684c6197c3492"
server_key_iv: "dfab030e988821d41292908092f9b790"
server_key_version: "encryptionkey@0"
I20260927 10:58:32.232235 19077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:32.234320 19091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:32.235402 19077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:32.235527 19077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "b35c8c88e67b40ed9f6c011013db2952"
format_stamp: "Formatted at 2026-09-27 10:58:32 on dist-test-slave-cffh"
server_key: "012358e305830cc942f684c6197c3492"
server_key_iv: "dfab030e988821d41292908092f9b790"
server_key_version: "encryptionkey@0"
I20260927 10:58:32.235628 19077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:32.264096 19077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:32.264750 19077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:32.264943 19077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:32.265499 19077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:32.266366 19077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:32.266435 19077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:32.266502 19077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:32.266552 19077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:32.274914 19077 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:38725
I20260927 10:58:32.274950 19204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:38725 every 8 connection(s)
I20260927 10:58:32.275766 19077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:58:32.278944 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 19077
I20260927 10:58:32.279028 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:58:32.279317 1211 external_mini_cluster.cc:1428] Setting key 2b0972c92fa926e368dcaeec33561eb8
I20260927 10:58:32.284780 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.277503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:46461 (local address 127.1.46.254:45227)
0927 10:58:32.277663 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.277667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.278747 (+ 1080us) server_negotiation.cc:407] Connection header received
0927 10:58:32.280395 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.280398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.280452 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.280534 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:32.281482 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.282045 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.283185 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.283390 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.283466 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.284398 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:32.284401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:32.284402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:32.284413 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:32.284429 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:32.284436 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:32.284490 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.284619 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.284675 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:58:32.285570 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.277880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45227 (local address 127.1.46.195:46461)
0927 10:58:32.278534 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.278619 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:32.280169 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:32.280753 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:32.280759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:32.280841 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:32.281320 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.281337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.282150 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.282152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.283087 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.283096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.283506 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.283509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.283636 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.284173 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:32.284174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:32.284210 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:32.284277 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:32.284503 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:32.284509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:32.284512 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:58:32.284603 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:58:32.284852 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
I20260927 10:58:32.286665 19205 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45227
I20260927 10:58:32.286857 19205 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:32.287283 19205 heartbeater.cc:511] Master 127.1.46.254:45227 requested a full tablet report, sending...
I20260927 10:58:32.288182 18755 ts_manager.cc:194] Registered new tserver with Master: b35c8c88e67b40ed9f6c011013db2952 (127.1.46.195:38725)
I20260927 10:58:32.288702 18755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.195:46461
I20260927 10:58:32.292929 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:58:32.298440 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.294797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.1.46.254:45227)
0927 10:58:32.294931 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.294935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.295080 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:58:32.295233 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.295237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.295278 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.295354 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:32.295633 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.296101 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.296902 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.297105 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.297210 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.298006 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:32.298010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:32.298014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:32.298033 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:32.298047 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:32.298055 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:32.298128 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.298267 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.298313 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:58:32.300985 18755 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:58:32.303797 18755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:58:32.319314 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.312982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45607 (local address 127.0.0.1:52700)
0927 10:58:32.313728 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.313742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:32.313817 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:32.314526 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:32.314533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:32.314558 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:32.314707 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.314716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.316271 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.316275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.317623 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.317634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.318659 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.318663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.318776 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.318832 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 10:58:32.318930 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:58:32.319161 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":52,"thread_start_us":123,"threads_started":1}
I20260927 10:58:32.319667 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.313530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.1.46.193:45607)
0927 10:58:32.314029 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.314043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.314062 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:58:32.314132 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.314136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.314351 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.314526 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:32.314839 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.316135 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.317766 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.318443 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.318549 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.319314 (+ 765us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.319375 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.319495 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":132,"threads_started":1}
I20260927 10:58:32.319936 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.313021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.1.46.195:38725)
0927 10:58:32.313530 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.313542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.313568 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:58:32.313806 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.313813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.314029 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.314223 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:32.314600 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.315860 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.317701 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.318519 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.318654 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.318825 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.319631 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.319741 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":178,"threads_started":1}
I20260927 10:58:32.319999 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.312990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.1.46.194:34021)
0927 10:58:32.314030 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.314045 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.314074 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:58:32.314163 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.314172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.314384 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.314621 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:32.314987 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.316219 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.317491 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.318432 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.318616 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.318825 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.319442 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.319800 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":139,"threads_started":1}
I20260927 10:58:32.320297 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.312269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:34021 (local address 127.0.0.1:59400)
0927 10:58:32.313389 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.313440 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:32.313570 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:32.314735 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:32.314738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:32.314757 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:32.314842 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.314848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.316385 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.316388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.317341 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.317351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.318900 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.318905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.319190 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.319216 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:58:32.319296 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:58:32.320147 (+ 851us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":900,"thread_start_us":63,"threads_started":1}
I20260927 10:58:32.320605 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.312277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:38725 (local address 127.0.0.1:46144)
0927 10:58:32.313464 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.313482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:32.313590 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:32.314241 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:32.314248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:32.314272 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:32.314419 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.314431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.315974 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.315981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.317554 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.317565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.318900 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.318905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.319097 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.319190 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 10:58:32.319283 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:58:32.319394 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":114,"thread_start_us":109,"threads_started":1}
I20260927 10:58:32.322954 18877 tablet_service.cc:1467] Processing CreateTablet for tablet 02095e6e56fa4c5686422145d01d53a4 (DEFAULT_TABLE table=test-table [id=ce7ffd6c7b6a4d1f86d19a6709fe61ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:32.323236 19008 tablet_service.cc:1467] Processing CreateTablet for tablet 02095e6e56fa4c5686422145d01d53a4 (DEFAULT_TABLE table=test-table [id=ce7ffd6c7b6a4d1f86d19a6709fe61ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:32.323328 19139 tablet_service.cc:1467] Processing CreateTablet for tablet 02095e6e56fa4c5686422145d01d53a4 (DEFAULT_TABLE table=test-table [id=ce7ffd6c7b6a4d1f86d19a6709fe61ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:32.324150 19008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02095e6e56fa4c5686422145d01d53a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:32.324150 18877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02095e6e56fa4c5686422145d01d53a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:32.324422 19139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02095e6e56fa4c5686422145d01d53a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:32.330940 19222 tablet_bootstrap.cc:492] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Bootstrap starting.
I20260927 10:58:32.332655 19224 tablet_bootstrap.cc:492] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733: Bootstrap starting.
I20260927 10:58:32.334511 19222 tablet_bootstrap.cc:654] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:32.334901 19223 tablet_bootstrap.cc:492] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Bootstrap starting.
I20260927 10:58:32.335037 19224 tablet_bootstrap.cc:654] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:32.335479 19222 log.cc:824] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:32.335933 19224 log.cc:824] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:32.336961 19223 tablet_bootstrap.cc:654] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:32.337644 19222 tablet_bootstrap.cc:492] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: No bootstrap required, opened a new log
I20260927 10:58:32.337858 19223 log.cc:824] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:32.337904 19222 ts_tablet_manager.cc:1397] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260927 10:58:32.338822 19224 tablet_bootstrap.cc:492] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733: No bootstrap required, opened a new log
I20260927 10:58:32.339012 19224 ts_tablet_manager.cc:1397] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260927 10:58:32.339900 19223 tablet_bootstrap.cc:492] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: No bootstrap required, opened a new log
I20260927 10:58:32.340092 19223 ts_tablet_manager.cc:1397] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:58:32.342976 19222 raft_consensus.cc:348] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.343271 19222 raft_consensus.cc:374] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:32.343369 19222 raft_consensus.cc:729] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c8c88e67b40ed9f6c011013db2952, State: Initialized, Role: FOLLOWER
I20260927 10:58:32.343451 19224 raft_consensus.cc:348] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.343719 19224 raft_consensus.cc:374] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:32.343820 19224 raft_consensus.cc:729] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b891f299908b4a898313d9260b167733, State: Initialized, Role: FOLLOWER
I20260927 10:58:32.343694 19223 raft_consensus.cc:348] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.343874 19222 consensus_queue.cc:260] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.343995 19223 raft_consensus.cc:374] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:32.344084 19223 raft_consensus.cc:729] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9b9e0280fe41f193518cf040e65428, State: Initialized, Role: FOLLOWER
I20260927 10:58:32.344259 19224 consensus_queue.cc:260] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.344539 19223 consensus_queue.cc:260] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.344957 19222 ts_tablet_manager.cc:1428] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:58:32.345220 19224 ts_tablet_manager.cc:1428] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260927 10:58:32.345223 19074 heartbeater.cc:503] Master 127.1.46.254:45227 was elected leader, sending a full tablet report...
I20260927 10:58:32.345361 19205 heartbeater.cc:503] Master 127.1.46.254:45227 was elected leader, sending a full tablet report...
I20260927 10:58:32.345546 18943 heartbeater.cc:503] Master 127.1.46.254:45227 was elected leader, sending a full tablet report...
I20260927 10:58:32.346484 19223 ts_tablet_manager.cc:1428] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 10:58:32.403312 18944 tablet.cc:2369] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:32.526800 19206 tablet.cc:2369] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:32.532075 19229 raft_consensus.cc:482] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:32.532315 19229 raft_consensus.cc:504] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.533506 19229 leader_election.cc:288] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e9b9e0280fe41f193518cf040e65428 (127.1.46.194:34021), b35c8c88e67b40ed9f6c011013db2952 (127.1.46.195:38725)
I20260927 10:58:32.537504 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.533856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:56085 (local address 127.1.46.195:38725)
0927 10:58:32.534051 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.534059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.534311 (+ 252us) server_negotiation.cc:407] Connection header received
0927 10:58:32.534465 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.534468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.534511 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.534612 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:32.534946 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.535383 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.536550 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.536924 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.537041 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.537090 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.537325 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.537386 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 10:58:32.537518 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.533825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:35581 (local address 127.1.46.194:34021)
0927 10:58:32.533962 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.533969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.534132 (+ 163us) server_negotiation.cc:407] Connection header received
0927 10:58:32.534292 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.534296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.534349 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.534436 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:32.534754 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.535361 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.536446 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.536932 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.537070 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.537120 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.537327 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.537394 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:58:32.537528 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.533713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:34021 (local address 127.1.46.193:35581)
0927 10:58:32.534023 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.534037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:32.534150 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:32.534460 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:32.534463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:32.534481 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:32.534639 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.534647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.535485 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.535488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.536299 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.536307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.537075 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.537079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.537207 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.537223 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:58:32.537308 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:58:32.537372 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":41,"thread_start_us":71,"threads_started":1}
I20260927 10:58:32.537539 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.533754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:38725 (local address 127.1.46.193:56085)
0927 10:58:32.534189 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.534222 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:32.534327 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:32.534666 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:32.534670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:32.534715 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:32.534838 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.534845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.535537 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.535540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.536421 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:32.536433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.537051 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.537055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:32.537189 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.537218 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:58:32.537311 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:58:32.537385 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":18,"thread_start_us":90,"threads_started":1}
I20260927 10:58:32.538173 19028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02095e6e56fa4c5686422145d01d53a4" candidate_uuid: "b891f299908b4a898313d9260b167733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9b9e0280fe41f193518cf040e65428" is_pre_election: true
I20260927 10:58:32.538447 19028 raft_consensus.cc:2454] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b891f299908b4a898313d9260b167733 in term 0.
I20260927 10:58:32.538448 19159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02095e6e56fa4c5686422145d01d53a4" candidate_uuid: "b891f299908b4a898313d9260b167733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c8c88e67b40ed9f6c011013db2952" is_pre_election: true
I20260927 10:58:32.538720 19159 raft_consensus.cc:2454] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b891f299908b4a898313d9260b167733 in term 0.
I20260927 10:58:32.538981 18830 leader_election.cc:302] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e9b9e0280fe41f193518cf040e65428, b891f299908b4a898313d9260b167733; no voters:
I20260927 10:58:32.539301 19229 raft_consensus.cc:2781] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:58:32.539389 19229 raft_consensus.cc:482] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:58:32.539422 19229 raft_consensus.cc:3037] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:32.540545 19229 raft_consensus.cc:504] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.540912 19229 leader_election.cc:288] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [CANDIDATE]: Term 1 election: Requested vote from peers 8e9b9e0280fe41f193518cf040e65428 (127.1.46.194:34021), b35c8c88e67b40ed9f6c011013db2952 (127.1.46.195:38725)
I20260927 10:58:32.541285 19028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02095e6e56fa4c5686422145d01d53a4" candidate_uuid: "b891f299908b4a898313d9260b167733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e9b9e0280fe41f193518cf040e65428"
I20260927 10:58:32.541388 19028 raft_consensus.cc:3037] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:32.541600 19159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02095e6e56fa4c5686422145d01d53a4" candidate_uuid: "b891f299908b4a898313d9260b167733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c8c88e67b40ed9f6c011013db2952"
I20260927 10:58:32.541735 19159 raft_consensus.cc:3037] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:32.542834 19028 raft_consensus.cc:2454] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b891f299908b4a898313d9260b167733 in term 1.
I20260927 10:58:32.543155 19159 raft_consensus.cc:2454] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b891f299908b4a898313d9260b167733 in term 1.
I20260927 10:58:32.543172 18830 leader_election.cc:302] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e9b9e0280fe41f193518cf040e65428, b891f299908b4a898313d9260b167733; no voters:
I20260927 10:58:32.543381 19229 raft_consensus.cc:2781] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:32.543645 19229 raft_consensus.cc:686] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [term 1 LEADER]: Becoming Leader. State: Replica: b891f299908b4a898313d9260b167733, State: Running, Role: LEADER
I20260927 10:58:32.544057 19229 consensus_queue.cc:237] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } }
I20260927 10:58:32.547175 18754 catalog_manager.cc:5495] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 reported cstate change: term changed from 0 to 1, leader changed from <none> to b891f299908b4a898313d9260b167733 (127.1.46.193). New cstate: current_term: 1 leader_uuid: "b891f299908b4a898313d9260b167733" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b891f299908b4a898313d9260b167733" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 45607 } health_report { overall_health: HEALTHY } } }
W20260927 10:58:32.587430 19075 tablet.cc:2369] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:32.611199 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.607770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.1.46.193:45607)
0927 10:58:32.607900 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:32.607904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:32.607917 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:58:32.608070 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:32.608072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:32.608130 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:32.608232 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:58:32.608493 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.608981 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.609814 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:32.610023 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:32.610114 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:32.610428 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:58:32.610882 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:58:32.610971 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:32.611042 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:58:32.611086 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:58:32.616784 19159 raft_consensus.cc:1261] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 1 FOLLOWER]: Refusing update from remote peer b891f299908b4a898313d9260b167733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:32.616817 19028 raft_consensus.cc:1261] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 1 FOLLOWER]: Refusing update from remote peer b891f299908b4a898313d9260b167733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:32.617434 19229 consensus_queue.cc:1035] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e9b9e0280fe41f193518cf040e65428" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:32.617648 19233 consensus_queue.cc:1035] T 02095e6e56fa4c5686422145d01d53a4 P b891f299908b4a898313d9260b167733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c8c88e67b40ed9f6c011013db2952" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:32.624660 19236 mvcc.cc:205] Tried to move back new op lower bound from 7333915494871732224 to 7333915494582800384. Current Snapshot: MvccSnapshot[applied={T|T < 7333915494871732224}]
I20260927 10:58:32.625839 19238 mvcc.cc:205] Tried to move back new op lower bound from 7333915494871732224 to 7333915494582800384. Current Snapshot: MvccSnapshot[applied={T|T < 7333915494871732224}]
I20260927 10:58:32.629037 19237 mvcc.cc:205] Tried to move back new op lower bound from 7333915494871732224 to 7333915494582800384. Current Snapshot: MvccSnapshot[applied={T|T < 7333915494871732224}]
I20260927 10:58:32.635535 18755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:32.635670 18755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:32.637985 18755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 520d934d68ed4957baf3c79076b4217a: Sending DeleteTablet for 3 replicas of tablet 02095e6e56fa4c5686422145d01d53a4
I20260927 10:58:32.638670 19139 tablet_service.cc:1514] Processing DeleteTablet for tablet 02095e6e56fa4c5686422145d01d53a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:32 UTC) from {username='slave'} at 127.0.0.1:46144
I20260927 10:58:32.638708 19008 tablet_service.cc:1514] Processing DeleteTablet for tablet 02095e6e56fa4c5686422145d01d53a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:32 UTC) from {username='slave'} at 127.0.0.1:59400
I20260927 10:58:32.639338 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 18815
I20260927 10:58:32.639400 19246 tablet_replica.cc:330] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: stopping tablet replica
I20260927 10:58:32.639389 18877 tablet_service.cc:1514] Processing DeleteTablet for tablet 02095e6e56fa4c5686422145d01d53a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:32 UTC) from {username='slave'} at 127.0.0.1:52700
I20260927 10:58:32.639655 19246 raft_consensus.cc:2229] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:32.639880 19246 raft_consensus.cc:2258] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:32.641304 19245 tablet_replica.cc:330] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: stopping tablet replica
I20260927 10:58:32.641474 19246 ts_tablet_manager.cc:1905] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:32.641628 19245 raft_consensus.cc:2229] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:32.641932 19245 raft_consensus.cc:2258] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:32.643775 19245 ts_tablet_manager.cc:1905] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:32.645010 19246 ts_tablet_manager.cc:1918] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:32.645112 19246 log.cc:1196] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/02095e6e56fa4c5686422145d01d53a4
I20260927 10:58:32.645419 19246 ts_tablet_manager.cc:1939] T 02095e6e56fa4c5686422145d01d53a4 P b35c8c88e67b40ed9f6c011013db2952: Deleting consensus metadata
I20260927 10:58:32.646090 19245 ts_tablet_manager.cc:1918] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:32.646188 19245 log.cc:1196] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/02095e6e56fa4c5686422145d01d53a4
I20260927 10:58:32.646625 18741 catalog_manager.cc:4841] TS b35c8c88e67b40ed9f6c011013db2952 (127.1.46.195:38725): tablet 02095e6e56fa4c5686422145d01d53a4 (table test-table [id=ce7ffd6c7b6a4d1f86d19a6709fe61ad]) successfully deleted
I20260927 10:58:32.646842 19245 ts_tablet_manager.cc:1939] T 02095e6e56fa4c5686422145d01d53a4 P 8e9b9e0280fe41f193518cf040e65428: Deleting consensus metadata
W20260927 10:58:32.647011 18739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:45607 (error 108)
I20260927 10:58:32.647269 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 18946
I20260927 10:58:32.651543 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.651314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:45607 (local address 127.0.0.1:40892)
0927 10:58:32.651428 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.651482 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:45607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:58:32.651703 18739 catalog_manager.cc:4584] TS b891f299908b4a898313d9260b167733 (127.1.46.193:45607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02095e6e56fa4c5686422145d01d53a4: Network error: Client connection negotiation failed: client connection to 127.1.46.193:45607: connect: Connection refused (error 111)
I20260927 10:58:32.652633 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 19077
I20260927 10:58:32.653831 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:32.652762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:34021 (local address 127.0.0.1:50620)
0927 10:58:32.653675 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:32.653738 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.194:34021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":827}
I20260927 10:58:32.657869 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 18724
2026-09-27T10:58:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1250 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:32 dist-test-slave-cffh krb5kdc[19250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:32 dist-test-slave-cffh krb5kdc[19250](info): set up 2 sockets
Sep 27 10:58:32 dist-test-slave-cffh krb5kdc[19250](info): commencing operation
krb5kdc: starting...
W20260927 10:58:33.726511 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:33 dist-test-slave-cffh krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:58:33.841229 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36973
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:33.947355 19263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:33.947598 19263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:33.947641 19263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:33.951581 19263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:33.951638 19263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:33.951656 19263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:33.951673 19263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:33.951687 19263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:33.956964 19263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:36973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:33.957935 19263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:33.958974 19263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:33.967455 19268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:33.967480 19269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:33.967741 19263 server_base.cc:1034] running on GCE node
W20260927 10:58:33.967480 19271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:33.968412 19263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:33.969461 19263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:33.970674 19263 hybrid_clock.cc:648] HybridClock initialized: now 1790506713970624 us; error 55 us; skew 500 ppm
Sep 27 10:58:33 dist-test-slave-cffh krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506713, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:33.973299 19263 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:58:33.974159 19263 webserver.cc:460] Webserver started at http://127.1.46.254:42827/ using document root <none> and password file <none>
I20260927 10:58:33.974808 19263 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:33.974901 19263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:33.975127 19263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:33.977174 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "5e8e4b19a47e47e1b60f84b2b696959a"
format_stamp: "Formatted at 2026-09-27 10:58:33 on dist-test-slave-cffh"
server_key: "18f13bb1a7fbdebae26a2fa9969fddd6"
server_key_iv: "837f63697a9c56ec1fce4e0188bd91cb"
server_key_version: "encryptionkey@0"
I20260927 10:58:33.977720 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "5e8e4b19a47e47e1b60f84b2b696959a"
format_stamp: "Formatted at 2026-09-27 10:58:33 on dist-test-slave-cffh"
server_key: "18f13bb1a7fbdebae26a2fa9969fddd6"
server_key_iv: "837f63697a9c56ec1fce4e0188bd91cb"
server_key_version: "encryptionkey@0"
I20260927 10:58:33.981778 19263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 10:58:33.984146 19278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:33.985389 19263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:58:33.985534 19263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "5e8e4b19a47e47e1b60f84b2b696959a"
format_stamp: "Formatted at 2026-09-27 10:58:33 on dist-test-slave-cffh"
server_key: "18f13bb1a7fbdebae26a2fa9969fddd6"
server_key_iv: "837f63697a9c56ec1fce4e0188bd91cb"
server_key_version: "encryptionkey@0"
I20260927 10:58:33.985672 19263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:34.011724 19263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:34.014468 19263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:34.014695 19263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:34.021529 19263 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:36973
I20260927 10:58:34.021553 19330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:36973 every 8 connection(s)
I20260927 10:58:34.022434 19263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:34.025053 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:34.026553 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 19263
I20260927 10:58:34.026731 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:34.027040 1211 external_mini_cluster.cc:1428] Setting key 32db119b8dd1f490c8400583bcb5f7fc
I20260927 10:58:34.031106 19331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a: Bootstrap starting.
I20260927 10:58:34.033186 19331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:34.033951 19331 log.cc:824] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:58:34 dist-test-slave-cffh krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:34.036075 19331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a: No bootstrap required, opened a new log
I20260927 10:58:34.040184 19331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36973 } }
I20260927 10:58:34.040385 19331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:34.040443 19331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e4b19a47e47e1b60f84b2b696959a, State: Initialized, Role: FOLLOWER
I20260927 10:58:34.040844 19331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36973 } }
I20260927 10:58:34.040885 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.028397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.1.46.254:36973)
0927 10:58:34.028781 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.028795 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.028827 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:58:34.029478 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.029492 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.029804 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.030119 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:34.030969 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.031962 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.032802 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.033279 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.033450 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.036257 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:34.036262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:34.036274 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:34.036298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:34.037925 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.038429 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.038433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.038437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.038536 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.038849 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.038851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.038853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.039230 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:34.039384 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.039750 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.040067 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260927 10:58:34.040964 19331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:34.041040 19331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:34.041175 19331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:34.041960 19331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36973 } }
I20260927 10:58:34.042284 19331 leader_election.cc:302] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e8e4b19a47e47e1b60f84b2b696959a; no voters:
I20260927 10:58:34.042626 19331 leader_election.cc:288] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:34.042706 19336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:34.043186 19336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [term 1 LEADER]: Becoming Leader. State: Replica: 5e8e4b19a47e47e1b60f84b2b696959a, State: Running, Role: LEADER
I20260927 10:58:34.043493 19336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36973 } }
I20260927 10:58:34.043829 19331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:34.044929 19338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e8e4b19a47e47e1b60f84b2b696959a. Latest consensus state: current_term: 1 leader_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36973 } } }
I20260927 10:58:34.045053 19338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:34.045428 19345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:34.045415 19337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e4b19a47e47e1b60f84b2b696959a" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 36973 } } }
I20260927 10:58:34.045559 19337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:34.048349 19345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:34.053444 19345 catalog_manager.cc:1347] Generated new cluster ID: 6c502304c15e4e239597b16d33b82611
I20260927 10:58:34.053550 19345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:34.062664 19345 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:34.063515 19345 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:34.072906 19345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a: Generated new TSK 0
I20260927 10:58:34.073558 19345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:58:34.099893 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36973
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:34.208101 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:34.208338 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:34.208379 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:34.212167 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:34.212212 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:34.212266 19357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:58:34.217229 19357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:34.218133 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:34.219060 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:34.227849 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:34.227872 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:34.228022 19357 server_base.cc:1034] running on GCE node
W20260927 10:58:34.227913 19365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:34.228556 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:34.229197 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:34.230376 19357 hybrid_clock.cc:648] HybridClock initialized: now 1790506714230344 us; error 47 us; skew 500 ppm
Sep 27 10:58:34 dist-test-slave-cffh krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506714, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:34.236606 19357 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:58:34.237470 19357 webserver.cc:460] Webserver started at http://127.1.46.193:44319/ using document root <none> and password file <none>
I20260927 10:58:34.238196 19357 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:34.238276 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:34.238528 19357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:34.240514 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "ddb078efee624f9b95f5a5494b870fb7"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "0c77dc3262cf7c7b3369649fd88ae901"
server_key_iv: "2f3b43ab48321220935a104ac7cd096c"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.241025 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "ddb078efee624f9b95f5a5494b870fb7"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "0c77dc3262cf7c7b3369649fd88ae901"
server_key_iv: "2f3b43ab48321220935a104ac7cd096c"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.244719 19357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 10:58:34.247121 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.248243 19357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:34.248400 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "ddb078efee624f9b95f5a5494b870fb7"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "0c77dc3262cf7c7b3369649fd88ae901"
server_key_iv: "2f3b43ab48321220935a104ac7cd096c"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.248505 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:34.269495 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:34.272670 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:34.272871 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:34.273437 19357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:34.274204 19357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:34.274271 19357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.274340 19357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:34.274382 19357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.283066 19357 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:39919
I20260927 10:58:34.283092 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:39919 every 8 connection(s)
I20260927 10:58:34.283916 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:58:34.285816 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 19357
I20260927 10:58:34.285955 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:58:34.286268 1211 external_mini_cluster.cc:1428] Setting key 265df61848e5565119434eb5f2a0c32b
Sep 27 10:58:34 dist-test-slave-cffh krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506714, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
I20260927 10:58:34.297741 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.285744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:41019 (local address 127.1.46.254:36973)
0927 10:58:34.285900 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.285906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.286897 (+ 991us) server_negotiation.cc:407] Connection header received
0927 10:58:34.288020 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.288023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.288073 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.288161 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:34.289725 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.290333 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.291598 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.291854 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.291967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.294290 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:34.294294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:34.294297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:34.294324 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:34.295942 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.296406 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.296409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.296410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.296478 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.296775 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.296777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.296778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.296932 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:34.297004 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.297477 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.297593 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:58:34.298702 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.286213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36973 (local address 127.1.46.193:41019)
0927 10:58:34.286755 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:34.286809 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:34.287806 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:34.288295 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:34.288301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:34.288937 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:34.289538 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.289561 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.290452 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.290457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.291473 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.291482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.291999 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.292003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.292176 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.292743 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:34.292759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:34.294132 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:34.296068 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:34.296072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:34.296083 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:34.296320 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:34.296557 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:34.296559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:34.296560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:34.296687 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:34.297021 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:34.297025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:34.297283 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 10:58:34.297431 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 10:58:34.297731 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260927 10:58:34.300011 19486 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36973
I20260927 10:58:34.300274 19486 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:34.300825 19486 heartbeater.cc:511] Master 127.1.46.254:36973 requested a full tablet report, sending...
I20260927 10:58:34.302320 19295 ts_manager.cc:194] Registered new tserver with Master: ddb078efee624f9b95f5a5494b870fb7 (127.1.46.193:39919)
I20260927 10:58:34.303436 19295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:41019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:58:34.313390 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36973
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:34.421731 19491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:34.421958 19491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:34.422040 19491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:34.425873 19491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:34.425922 19491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:34.426036 19491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:58:34.431049 19491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:34.431949 19491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:34.432873 19491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:58:34.442022 19491 server_base.cc:1034] running on GCE node
W20260927 10:58:34.441959 19496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:34.441974 19499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:34.441990 19497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:34.442543 19491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:34.443233 19491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:34.444429 19491 hybrid_clock.cc:648] HybridClock initialized: now 1790506714444412 us; error 28 us; skew 500 ppm
Sep 27 10:58:34 dist-test-slave-cffh krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506714, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:34.447175 19491 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:58:34.447875 19491 webserver.cc:460] Webserver started at http://127.1.46.194:45141/ using document root <none> and password file <none>
I20260927 10:58:34.448367 19491 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:34.448423 19491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:34.448645 19491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:34.450470 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "61d350f5999503ce91ffd7af8bf68bd6"
server_key_iv: "66536f6343bb7dffba3f228588cd89bb"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.451089 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "61d350f5999503ce91ffd7af8bf68bd6"
server_key_iv: "66536f6343bb7dffba3f228588cd89bb"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.454511 19491 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:34.456815 19506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.457906 19491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:58:34.458015 19491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "61d350f5999503ce91ffd7af8bf68bd6"
server_key_iv: "66536f6343bb7dffba3f228588cd89bb"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.458110 19491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:34.487156 19491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:34.490284 19491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:34.490494 19491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:34.491057 19491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:34.491822 19491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:34.491863 19491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.491926 19491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:34.491966 19491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.500222 19491 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:42181
I20260927 10:58:34.500253 19619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:42181 every 8 connection(s)
I20260927 10:58:34.501051 19491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:58:34.508951 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 19491
I20260927 10:58:34.509080 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:58:34.509436 1211 external_mini_cluster.cc:1428] Setting key 4bf97adfb3bf29e4bbd5fd85a1dca1fc
Sep 27 10:58:34 dist-test-slave-cffh krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506714, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:34.514330 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.502615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:35663 (local address 127.1.46.254:36973)
0927 10:58:34.502748 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.502751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.503544 (+ 793us) server_negotiation.cc:407] Connection header received
0927 10:58:34.504750 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.504754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.504812 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.504884 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:34.505934 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.506470 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.507602 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.507838 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.507950 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.510476 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:34.510485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:34.510487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:34.510509 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:34.512134 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.512716 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.512719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.512720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.512791 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.513236 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.513238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.513239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.513395 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:34.513468 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.514125 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.514220 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:58:34.515532 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.502981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36973 (local address 127.1.46.194:35663)
0927 10:58:34.503404 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:34.503443 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:34.504542 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:34.505038 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:34.505044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:34.505371 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:34.505798 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.505810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.506614 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.506618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.507479 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.507488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.507981 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.507985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.508156 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.508659 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:34.508676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:34.510317 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:34.512266 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:34.512270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:34.512280 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:34.512593 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:34.512931 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:34.512936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:34.512940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:34.513135 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:34.513499 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:34.513507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:34.513866 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 10:58:34.514087 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 10:58:34.514436 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260927 10:58:34.516734 19620 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36973
I20260927 10:58:34.516952 19620 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:34.517413 19620 heartbeater.cc:511] Master 127.1.46.254:36973 requested a full tablet report, sending...
I20260927 10:58:34.518430 19295 ts_manager.cc:194] Registered new tserver with Master: 0b43a8f4f57d465eb4cdeb73baedf2f3 (127.1.46.194:42181)
I20260927 10:58:34.519030 19295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:35663
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:58:34.535836 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36973
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:34.644764 19625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:34.645049 19625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:34.645134 19625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:34.649272 19625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:34.649348 19625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:34.649434 19625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:58:34.654778 19625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:36973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:34.655706 19625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:34.656639 19625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:34.665617 19631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:34.665622 19630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:34.665732 19625 server_base.cc:1034] running on GCE node
W20260927 10:58:34.665622 19633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:34.666287 19625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:34.666904 19625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:34.668049 19625 hybrid_clock.cc:648] HybridClock initialized: now 1790506714668029 us; error 33 us; skew 500 ppm
Sep 27 10:58:34 dist-test-slave-cffh krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506714, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:34.670747 19625 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:58:34.671515 19625 webserver.cc:460] Webserver started at http://127.1.46.195:38579/ using document root <none> and password file <none>
I20260927 10:58:34.672029 19625 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:34.672086 19625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:34.672279 19625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:34.674129 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "4748c659ccb8448cbe071d32455b6556"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "e450cf02a2caca965600d58407cc943e"
server_key_iv: "7ab6a77edc24f4c91814bf057c8806a9"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.674708 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "4748c659ccb8448cbe071d32455b6556"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "e450cf02a2caca965600d58407cc943e"
server_key_iv: "7ab6a77edc24f4c91814bf057c8806a9"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.678052 19625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:34.680372 19640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.681403 19625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:34.681510 19625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "4748c659ccb8448cbe071d32455b6556"
format_stamp: "Formatted at 2026-09-27 10:58:34 on dist-test-slave-cffh"
server_key: "e450cf02a2caca965600d58407cc943e"
server_key_iv: "7ab6a77edc24f4c91814bf057c8806a9"
server_key_version: "encryptionkey@0"
I20260927 10:58:34.681620 19625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:34.709151 19625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:34.712013 19625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:34.712215 19625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:34.712831 19625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:34.713603 19625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:34.713647 19625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.713721 19625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:34.713759 19625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:34.722486 19625 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:37381
I20260927 10:58:34.722560 19753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:37381 every 8 connection(s)
I20260927 10:58:34.723373 19625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:58:34.731122 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 19625
I20260927 10:58:34.731256 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:58:34.731547 1211 external_mini_cluster.cc:1428] Setting key ce7ae52888e0e0bc7c2affae2de6be14
Sep 27 10:58:34 dist-test-slave-cffh krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506714, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:34.736356 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.725099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:50269 (local address 127.1.46.254:36973)
0927 10:58:34.725238 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.725241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.726230 (+ 989us) server_negotiation.cc:407] Connection header received
0927 10:58:34.727105 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.727109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.727167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.727269 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:34.728352 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.728891 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.729991 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.730213 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.730326 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.732995 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:34.732997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:34.732999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:34.733021 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:34.734335 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.734885 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.734890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.734893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.734972 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.735353 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.735357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.735359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.735522 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:34.735622 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.736078 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.736186 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:58:34.736997 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.725412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:36973 (local address 127.1.46.195:50269)
0927 10:58:34.726063 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:34.726116 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:34.726906 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:34.727428 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:34.727434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:34.727765 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:34.728196 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.728208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.729018 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.729023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.729866 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.729874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.730356 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.730360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.730519 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.731158 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:34.731179 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:34.732819 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:34.734449 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:34.734455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:34.734468 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:34.734786 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:34.735070 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:34.735074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:34.735077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:34.735221 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:34.735632 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:34.735640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:34.735841 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 10:58:34.736023 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 10:58:34.736320 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":123,"threads_started":1}
I20260927 10:58:34.737936 19754 heartbeater.cc:347] Connected to a master server at 127.1.46.254:36973
I20260927 10:58:34.738183 19754 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:34.738683 19754 heartbeater.cc:511] Master 127.1.46.254:36973 requested a full tablet report, sending...
I20260927 10:58:34.739604 19295 ts_manager.cc:194] Registered new tserver with Master: 4748c659ccb8448cbe071d32455b6556 (127.1.46.195:37381)
I20260927 10:58:34.740154 19295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:50269
I20260927 10:58:34.745100 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:58:34.754340 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.747052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.1.46.254:36973)
0927 10:58:34.747165 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.747168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.747321 (+ 153us) server_negotiation.cc:407] Connection header received
0927 10:58:34.747414 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.747416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.747447 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.747504 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:34.748132 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.748605 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.749526 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.749734 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.749812 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.751230 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:34.751233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:34.751235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:34.751254 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:34.752734 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.753197 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.753199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.753201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.753266 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:34.753576 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:34.753578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:34.753579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:34.753745 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:34.753853 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.754066 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.754165 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:58:34.757014 19295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:58:34.760002 19295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:58:34.773674 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.768071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:42181 (local address 127.0.0.1:34780)
0927 10:58:34.768569 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:34.768628 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:34.768794 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:34.769579 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:34.769585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:34.769609 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:34.769734 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.769744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.771017 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.771023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.772156 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.772166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.773178 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.773182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.773282 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.773316 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:58:34.773419 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:58:34.773531 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":64,"threads_started":1}
I20260927 10:58:34.773684 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.768220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.1.46.194:42181)
0927 10:58:34.768695 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.768708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.768732 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:58:34.768994 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.769000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.769230 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.769380 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:34.769859 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.770900 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.772286 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.773056 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.773181 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.773321 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.773447 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.773531 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":145,"threads_started":1}
I20260927 10:58:34.774133 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.768242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39919 (local address 127.0.0.1:33222)
0927 10:58:34.769040 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:34.769054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:34.769140 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:34.769810 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:34.769813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:34.769829 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:34.769928 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.769934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.771595 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.771598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.772703 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.772712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.773722 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.773726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.773832 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.773853 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:58:34.773902 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 10:58:34.773982 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":28,"thread_start_us":109,"threads_started":1}
I20260927 10:58:34.774468 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.768762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:37381 (local address 127.0.0.1:56560)
0927 10:58:34.769362 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:34.769374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:34.769452 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:34.770149 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:34.770152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:34.770171 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:34.770271 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.770277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.771669 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.771673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.772617 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:34.772628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.773722 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.773726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:34.774153 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.774170 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:58:34.774272 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 10:58:34.774336 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":334,"thread_start_us":88,"threads_started":1}
I20260927 10:58:34.774806 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.768990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.1.46.193:39919)
0927 10:58:34.769345 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.769359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.769383 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:58:34.769460 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.769465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.769649 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.770110 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:34.770250 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.771452 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.772796 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.773579 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.774386 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.774516 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.774559 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.774675 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260927 10:58:34.775027 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:34.768951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.1.46.195:37381)
0927 10:58:34.769692 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:34.769705 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:34.769729 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:58:34.769801 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:34.769806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:34.769985 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:34.770142 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:34.770395 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.771501 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.772750 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:34.773579 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:34.773772 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:34.773911 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:34.774757 (+ 846us) server_negotiation.cc:299] Negotiation successful
0927 10:58:34.774878 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":136,"threads_started":1}
I20260927 10:58:34.777038 19554 tablet_service.cc:1467] Processing CreateTablet for tablet af0608caf0664080aafc489e948cec91 (DEFAULT_TABLE table=test-table [id=d7dfbf1274f14df29b59cbda5b8e3127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:34.777243 19688 tablet_service.cc:1467] Processing CreateTablet for tablet af0608caf0664080aafc489e948cec91 (DEFAULT_TABLE table=test-table [id=d7dfbf1274f14df29b59cbda5b8e3127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:34.777237 19420 tablet_service.cc:1467] Processing CreateTablet for tablet af0608caf0664080aafc489e948cec91 (DEFAULT_TABLE table=test-table [id=d7dfbf1274f14df29b59cbda5b8e3127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:34.778093 19688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0608caf0664080aafc489e948cec91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:34.778110 19554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0608caf0664080aafc489e948cec91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:34.778108 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0608caf0664080aafc489e948cec91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:34.783632 19771 tablet_bootstrap.cc:492] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Bootstrap starting.
I20260927 10:58:34.785497 19773 tablet_bootstrap.cc:492] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: Bootstrap starting.
I20260927 10:58:34.786537 19771 tablet_bootstrap.cc:654] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:34.786906 19772 tablet_bootstrap.cc:492] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Bootstrap starting.
I20260927 10:58:34.787436 19773 tablet_bootstrap.cc:654] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:34.787683 19771 log.cc:824] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:34.788655 19773 log.cc:824] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:34.789373 19772 tablet_bootstrap.cc:654] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:34.789919 19771 tablet_bootstrap.cc:492] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: No bootstrap required, opened a new log
I20260927 10:58:34.790110 19771 ts_tablet_manager.cc:1397] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 10:58:34.790208 19772 log.cc:824] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:34.790913 19773 tablet_bootstrap.cc:492] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: No bootstrap required, opened a new log
I20260927 10:58:34.791142 19773 ts_tablet_manager.cc:1397] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:58:34.792092 19772 tablet_bootstrap.cc:492] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: No bootstrap required, opened a new log
I20260927 10:58:34.792274 19772 ts_tablet_manager.cc:1397] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:58:34.793287 19771 raft_consensus.cc:348] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:34.793514 19771 raft_consensus.cc:374] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:34.793582 19771 raft_consensus.cc:729] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4748c659ccb8448cbe071d32455b6556, State: Initialized, Role: FOLLOWER
I20260927 10:58:34.794018 19771 consensus_queue.cc:260] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:34.794600 19754 heartbeater.cc:503] Master 127.1.46.254:36973 was elected leader, sending a full tablet report...
I20260927 10:58:34.794958 19771 ts_tablet_manager.cc:1428] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:58:34.794837 19773 raft_consensus.cc:348] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:34.795085 19773 raft_consensus.cc:374] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:34.795130 19773 raft_consensus.cc:729] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb078efee624f9b95f5a5494b870fb7, State: Initialized, Role: FOLLOWER
I20260927 10:58:34.795506 19773 consensus_queue.cc:260] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:34.795934 19772 raft_consensus.cc:348] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:34.796124 19486 heartbeater.cc:503] Master 127.1.46.254:36973 was elected leader, sending a full tablet report...
I20260927 10:58:34.796221 19772 raft_consensus.cc:374] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:34.796283 19772 raft_consensus.cc:729] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b43a8f4f57d465eb4cdeb73baedf2f3, State: Initialized, Role: FOLLOWER
I20260927 10:58:34.796643 19772 consensus_queue.cc:260] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:34.797224 19773 ts_tablet_manager.cc:1428] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:58:34.797495 19772 ts_tablet_manager.cc:1428] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:58:34.797813 19620 heartbeater.cc:503] Master 127.1.46.254:36973 was elected leader, sending a full tablet report...
W20260927 10:58:34.974349 19755 tablet.cc:2369] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:35.002110 19621 tablet.cc:2369] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:35.035252 19487 tablet.cc:2369] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:35.121979 19778 raft_consensus.cc:482] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:35.122224 19778 raft_consensus.cc:504] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:35.123401 19778 leader_election.cc:288] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b43a8f4f57d465eb4cdeb73baedf2f3 (127.1.46.194:42181), 4748c659ccb8448cbe071d32455b6556 (127.1.46.195:37381)
I20260927 10:58:35.127260 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.123795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:50449 (local address 127.1.46.194:42181)
0927 10:58:35.123955 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:35.123959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:35.124097 (+ 138us) server_negotiation.cc:407] Connection header received
0927 10:58:35.124281 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:35.124285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:35.124337 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:35.124423 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:35.124708 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.125195 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.126264 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.126717 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.126839 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.126888 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:35.127062 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 10:58:35.127143 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 10:58:35.127279 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.123672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:42181 (local address 127.1.46.193:50449)
0927 10:58:35.123984 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:35.124001 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:35.124168 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:35.124431 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:35.124433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:35.124448 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:35.124594 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.124599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.125310 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.125315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.126144 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.126152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.126830 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.126833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.126921 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.126946 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:58:35.127037 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:58:35.127133 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":6,"spinlock_wait_cycles":2944,"thread_start_us":110,"threads_started":1}
I20260927 10:58:35.127473 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.123822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:44411 (local address 127.1.46.195:37381)
0927 10:58:35.124012 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:35.124017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:35.124148 (+ 131us) server_negotiation.cc:407] Connection header received
0927 10:58:35.124315 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:35.124319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:35.124360 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:35.124443 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:35.124718 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.125208 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.126261 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.126666 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.126791 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.126843 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:35.127288 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 10:58:35.127354 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 10:58:35.127854 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.123689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:37381 (local address 127.1.46.193:44411)
0927 10:58:35.124043 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:35.124058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:35.124191 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:35.124455 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:35.124457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:35.124494 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:35.124609 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.124614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.125321 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.125327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.126140 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.126148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.126784 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.126787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.127135 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.127164 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:58:35.127619 (+ 455us) client_negotiation.cc:240] Negotiation successful
0927 10:58:35.127701 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":33,"spinlock_wait_cycles":1536,"thread_start_us":98,"threads_started":1}
I20260927 10:58:35.128010 19574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0608caf0664080aafc489e948cec91" candidate_uuid: "ddb078efee624f9b95f5a5494b870fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" is_pre_election: true
I20260927 10:58:35.128340 19574 raft_consensus.cc:2454] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb078efee624f9b95f5a5494b870fb7 in term 0.
I20260927 10:58:35.128505 19708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0608caf0664080aafc489e948cec91" candidate_uuid: "ddb078efee624f9b95f5a5494b870fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4748c659ccb8448cbe071d32455b6556" is_pre_election: true
I20260927 10:58:35.128783 19708 raft_consensus.cc:2454] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb078efee624f9b95f5a5494b870fb7 in term 0.
I20260927 10:58:35.128887 19374 leader_election.cc:302] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b43a8f4f57d465eb4cdeb73baedf2f3, ddb078efee624f9b95f5a5494b870fb7; no voters:
I20260927 10:58:35.129266 19778 raft_consensus.cc:2781] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:58:35.129349 19778 raft_consensus.cc:482] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:58:35.129392 19778 raft_consensus.cc:3037] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:35.130396 19778 raft_consensus.cc:504] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:35.130776 19778 leader_election.cc:288] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [CANDIDATE]: Term 1 election: Requested vote from peers 0b43a8f4f57d465eb4cdeb73baedf2f3 (127.1.46.194:42181), 4748c659ccb8448cbe071d32455b6556 (127.1.46.195:37381)
I20260927 10:58:35.131117 19574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0608caf0664080aafc489e948cec91" candidate_uuid: "ddb078efee624f9b95f5a5494b870fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3"
I20260927 10:58:35.131121 19708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0608caf0664080aafc489e948cec91" candidate_uuid: "ddb078efee624f9b95f5a5494b870fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4748c659ccb8448cbe071d32455b6556"
I20260927 10:58:35.131244 19574 raft_consensus.cc:3037] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:35.131254 19708 raft_consensus.cc:3037] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:35.132221 19708 raft_consensus.cc:2454] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb078efee624f9b95f5a5494b870fb7 in term 1.
I20260927 10:58:35.132241 19574 raft_consensus.cc:2454] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb078efee624f9b95f5a5494b870fb7 in term 1.
I20260927 10:58:35.132506 19373 leader_election.cc:302] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4748c659ccb8448cbe071d32455b6556, ddb078efee624f9b95f5a5494b870fb7; no voters:
I20260927 10:58:35.132707 19778 raft_consensus.cc:2781] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:35.132938 19778 raft_consensus.cc:686] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 1 LEADER]: Becoming Leader. State: Replica: ddb078efee624f9b95f5a5494b870fb7, State: Running, Role: LEADER
I20260927 10:58:35.133242 19778 consensus_queue.cc:237] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } }
I20260927 10:58:35.136368 19293 catalog_manager.cc:5495] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb078efee624f9b95f5a5494b870fb7 (127.1.46.193). New cstate: current_term: 1 leader_uuid: "ddb078efee624f9b95f5a5494b870fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb078efee624f9b95f5a5494b870fb7" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 39919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 } health_report { overall_health: UNKNOWN } } }
I20260927 10:58:35.224232 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.220966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.1.46.193:39919)
0927 10:58:35.221116 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:35.221120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:35.221155 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:58:35.221220 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:35.221223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:35.221279 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:35.221358 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:58:35.221569 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.222054 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.222901 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.223122 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.223201 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.223483 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:58:35.223878 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:58:35.223982 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:35.224082 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 10:58:35.224129 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:58:35.229897 19708 raft_consensus.cc:1261] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 1 FOLLOWER]: Refusing update from remote peer ddb078efee624f9b95f5a5494b870fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:35.229907 19574 raft_consensus.cc:1261] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 1 FOLLOWER]: Refusing update from remote peer ddb078efee624f9b95f5a5494b870fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:35.230616 19778 consensus_queue.cc:1035] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4748c659ccb8448cbe071d32455b6556" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:35.230831 19782 consensus_queue.cc:1035] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b43a8f4f57d465eb4cdeb73baedf2f3" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:35.239144 19787 mvcc.cc:205] Tried to move back new op lower bound from 7333915505574592512 to 7333915505187897344. Current Snapshot: MvccSnapshot[applied={T|T < 7333915505574592512}]
I20260927 10:58:35.239516 19786 mvcc.cc:205] Tried to move back new op lower bound from 7333915505574592512 to 7333915505187897344. Current Snapshot: MvccSnapshot[applied={T|T < 7333915505574592512 or (T in {7333915505574592512})}]
I20260927 10:58:35.245090 19295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:35.245265 19295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:35.247587 19295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e8e4b19a47e47e1b60f84b2b696959a: Sending DeleteTablet for 3 replicas of tablet af0608caf0664080aafc489e948cec91
I20260927 10:58:35.248340 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet af0608caf0664080aafc489e948cec91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:35 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:33222
I20260927 10:58:35.248340 19688 tablet_service.cc:1514] Processing DeleteTablet for tablet af0608caf0664080aafc489e948cec91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:35 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:56560
I20260927 10:58:35.248780 19796 tablet_replica.cc:330] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7: stopping tablet replica
I20260927 10:58:35.248767 19554 tablet_service.cc:1514] Processing DeleteTablet for tablet af0608caf0664080aafc489e948cec91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:35 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:34780
I20260927 10:58:35.249068 19796 raft_consensus.cc:2229] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:58:35.249212 19797 tablet_replica.cc:330] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: stopping tablet replica
I20260927 10:58:35.249481 19796 raft_consensus.cc:2258] T af0608caf0664080aafc489e948cec91 P ddb078efee624f9b95f5a5494b870fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:35.249540 19797 raft_consensus.cc:2229] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:35.249919 19797 raft_consensus.cc:2258] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:35.250186 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 19357
I20260927 10:58:35.251361 19797 ts_tablet_manager.cc:1905] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:35.252341 19795 tablet_replica.cc:330] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: stopping tablet replica
I20260927 10:58:35.252588 19795 raft_consensus.cc:2229] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:35.252878 19795 raft_consensus.cc:2258] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:35.253916 19795 ts_tablet_manager.cc:1905] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:35.254788 19797 ts_tablet_manager.cc:1918] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:35.254889 19797 log.cc:1196] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/af0608caf0664080aafc489e948cec91
I20260927 10:58:35.255256 19797 ts_tablet_manager.cc:1939] T af0608caf0664080aafc489e948cec91 P 0b43a8f4f57d465eb4cdeb73baedf2f3: Deleting consensus metadata
I20260927 10:58:35.256235 19795 ts_tablet_manager.cc:1918] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:35.256340 19795 log.cc:1196] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/af0608caf0664080aafc489e948cec91
I20260927 10:58:35.256449 19280 catalog_manager.cc:4841] TS 0b43a8f4f57d465eb4cdeb73baedf2f3 (127.1.46.194:42181): tablet af0608caf0664080aafc489e948cec91 (table test-table [id=d7dfbf1274f14df29b59cbda5b8e3127]) successfully deleted
I20260927 10:58:35.256631 19795 ts_tablet_manager.cc:1939] T af0608caf0664080aafc489e948cec91 P 4748c659ccb8448cbe071d32455b6556: Deleting consensus metadata
I20260927 10:58:35.257555 19279 catalog_manager.cc:4841] TS 4748c659ccb8448cbe071d32455b6556 (127.1.46.195:37381): tablet af0608caf0664080aafc489e948cec91 (table test-table [id=d7dfbf1274f14df29b59cbda5b8e3127]) successfully deleted
W20260927 10:58:35.258745 19279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:39919 (error 108)
I20260927 10:58:35.258812 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 19491
I20260927 10:58:35.263478 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.263150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:39919 (local address 127.0.0.1:33248)
0927 10:58:35.263301 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:35.263381 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:58:35.263667 19279 catalog_manager.cc:4584] TS ddb078efee624f9b95f5a5494b870fb7 (127.1.46.193:39919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af0608caf0664080aafc489e948cec91: Network error: Client connection negotiation failed: client connection to 127.1.46.193:39919: connect: Connection refused (error 111)
I20260927 10:58:35.265216 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 19625
I20260927 10:58:35.271221 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 19263
2026-09-27T10:58:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2614 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T10:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:35Z Disabled control of system clock
I20260927 10:58:35.302024 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:39915
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:39915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:58:35.408344 19802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:35.408601 19802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:35.408636 19802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:35.412537 19802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:35.412586 19802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:35.412602 19802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:35.412618 19802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:35.412633 19802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:35.417450 19802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:39915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:39915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:35.418447 19802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:35.419772 19802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:35.427881 19810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:35.427938 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:35.428042 19802 server_base.cc:1034] running on GCE node
W20260927 10:58:35.427881 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:35.428741 19802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:35.429760 19802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:35.430927 19802 hybrid_clock.cc:648] HybridClock initialized: now 1790506715430898 us; error 39 us; skew 500 ppm
I20260927 10:58:35.432787 19802 webserver.cc:460] Webserver started at http://127.1.46.254:36029/ using document root <none> and password file <none>
I20260927 10:58:35.433327 19802 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:35.433386 19802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:35.433601 19802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:35.435563 19802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "06ca98296b6648fba4f68f218093ba75"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "493b85f7e171725476e8eee9e22de977"
server_key_iv: "13abde25aa4fe2a7a7c22ec561ce1e3e"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.436102 19802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "06ca98296b6648fba4f68f218093ba75"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "493b85f7e171725476e8eee9e22de977"
server_key_iv: "13abde25aa4fe2a7a7c22ec561ce1e3e"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.439517 19802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:58:35.441694 19816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:35.442890 19802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:35.443015 19802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "06ca98296b6648fba4f68f218093ba75"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "493b85f7e171725476e8eee9e22de977"
server_key_iv: "13abde25aa4fe2a7a7c22ec561ce1e3e"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.443112 19802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:35.462726 19802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:35.463316 19802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:35.463495 19802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:35.469907 19802 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:39915
I20260927 10:58:35.469942 19868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:39915 every 8 connection(s)
I20260927 10:58:35.470770 19802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:35.473680 19869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:35.477353 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 19802
I20260927 10:58:35.477515 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:35.477836 1211 external_mini_cluster.cc:1428] Setting key 6311afddcb5b587e5cc2c4c3c807c35d
I20260927 10:58:35.479157 19869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75: Bootstrap starting.
I20260927 10:58:35.481714 19869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:35.482553 19869 log.cc:824] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:35.484431 19869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75: No bootstrap required, opened a new log
I20260927 10:58:35.488103 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.479307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.1.46.254:39915)
0927 10:58:35.479932 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:35.479950 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:35.479990 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:58:35.480823 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:35.480845 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:35.481303 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:35.481649 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:35.482111 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.483453 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.484331 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.484995 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.485157 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.485970 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:35.485976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:35.485994 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:35.486468 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:35.486531 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:35.486557 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:35.486694 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:35.486829 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 10:58:35.487085 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260927 10:58:35.488765 19869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ca98296b6648fba4f68f218093ba75" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39915 } }
I20260927 10:58:35.489013 19869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:35.489060 19869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ca98296b6648fba4f68f218093ba75, State: Initialized, Role: FOLLOWER
I20260927 10:58:35.489557 19869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ca98296b6648fba4f68f218093ba75" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39915 } }
I20260927 10:58:35.489714 19869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:35.489768 19869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:35.489856 19869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:35.490876 19869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ca98296b6648fba4f68f218093ba75" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39915 } }
I20260927 10:58:35.491150 19869 leader_election.cc:302] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06ca98296b6648fba4f68f218093ba75; no voters:
I20260927 10:58:35.491416 19869 leader_election.cc:288] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:35.491501 19874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:35.491683 19874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [term 1 LEADER]: Becoming Leader. State: Replica: 06ca98296b6648fba4f68f218093ba75, State: Running, Role: LEADER
I20260927 10:58:35.491963 19874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ca98296b6648fba4f68f218093ba75" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39915 } }
I20260927 10:58:35.492602 19869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:35.493773 19876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06ca98296b6648fba4f68f218093ba75. Latest consensus state: current_term: 1 leader_uuid: "06ca98296b6648fba4f68f218093ba75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ca98296b6648fba4f68f218093ba75" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39915 } } }
I20260927 10:58:35.493886 19876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:35.494092 19875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06ca98296b6648fba4f68f218093ba75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ca98296b6648fba4f68f218093ba75" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 39915 } } }
I20260927 10:58:35.494177 19875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:35.494270 19883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:35.497438 19883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:35.502125 19883 catalog_manager.cc:1347] Generated new cluster ID: 9bd72487597649d7881b04ec7f8a6eeb
I20260927 10:58:35.502188 19883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:35.513377 19883 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:35.514227 19883 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:35.518913 19883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75: Generated new TSK 0
I20260927 10:58:35.519531 19883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:35.523029 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39915
--builtin_ntp_servers=127.1.46.212:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:58:35.638202 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:35.638447 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:35.638485 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:35.642694 19893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:35.642747 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:35.642827 19893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:58:35.647724 19893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:35.648744 19893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:35.650137 19893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:35.659039 19901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:35.659015 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:35.659029 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:35.659384 19893 server_base.cc:1034] running on GCE node
I20260927 10:58:35.659940 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:35.660495 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:35.661688 19893 hybrid_clock.cc:648] HybridClock initialized: now 1790506715661675 us; error 25 us; skew 500 ppm
I20260927 10:58:35.663370 19893 webserver.cc:460] Webserver started at http://127.1.46.193:37943/ using document root <none> and password file <none>
I20260927 10:58:35.663905 19893 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:35.663990 19893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:35.664201 19893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:35.666134 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "f627a1c5594d4062960460739254017f"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "d460b3f8488153fd569c8f8517d0c030"
server_key_iv: "8fa99f4b181ff155dd90a61c52a85742"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.666743 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "f627a1c5594d4062960460739254017f"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "d460b3f8488153fd569c8f8517d0c030"
server_key_iv: "8fa99f4b181ff155dd90a61c52a85742"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.670163 19893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:58:35.672488 19907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:35.673640 19893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:35.673772 19893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "f627a1c5594d4062960460739254017f"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "d460b3f8488153fd569c8f8517d0c030"
server_key_iv: "8fa99f4b181ff155dd90a61c52a85742"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.673875 19893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:35.701695 19893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:35.702390 19893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:35.702636 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:35.703248 19893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:35.704169 19893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:35.704246 19893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:35.704319 19893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:35.704360 19893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:35.713280 19893 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:34707
I20260927 10:58:35.713320 20020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:34707 every 8 connection(s)
I20260927 10:58:35.714148 19893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:58:35.720676 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 19893
I20260927 10:58:35.720800 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:58:35.721104 1211 external_mini_cluster.cc:1428] Setting key fe4a99d262ab79d77cb6a5af3dfaea1a
I20260927 10:58:35.723030 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39915
--builtin_ntp_servers=127.1.46.212:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:58:35.723538 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.716031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:40439 (local address 127.1.46.254:39915)
0927 10:58:35.716194 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:35.716199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:35.717207 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 10:58:35.718561 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:35.718604 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:35.718668 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:35.718751 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:35.719579 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.720127 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.721367 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.721731 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.721852 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.722957 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:35.722961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:35.722965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:35.722978 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:35.723001 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:35.723015 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:35.723093 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:35.723342 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 10:58:35.723402 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:58:35.724326 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.716385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39915 (local address 127.1.46.193:40439)
0927 10:58:35.717056 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:35.717093 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:35.718359 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:35.718908 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:35.718914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:35.718994 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:35.719450 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.719463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.720236 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.720238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.721248 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.721257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.721887 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.721891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.722054 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.722708 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:35.722710 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:35.722748 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:35.722808 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:35.723095 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:35.723106 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:35.723115 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:58:35.723332 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 10:58:35.723595 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260927 10:58:35.725508 20021 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39915
I20260927 10:58:35.725742 20021 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:35.726243 20021 heartbeater.cc:511] Master 127.1.46.254:39915 requested a full tablet report, sending...
I20260927 10:58:35.727857 19833 ts_manager.cc:194] Registered new tserver with Master: f627a1c5594d4062960460739254017f (127.1.46.193:34707)
I20260927 10:58:35.729576 19833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.193:40439
W20260927 10:58:35.831243 20024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:35.831516 20024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:35.831554 20024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:35.835456 20024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:35.835505 20024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:35.835556 20024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:58:35.840101 20024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:35.841076 20024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:35.842352 20024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:35.851593 20032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:35.851714 20024 server_base.cc:1034] running on GCE node
W20260927 10:58:35.851639 20029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:35.851631 20030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:35.852276 20024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:35.852823 20024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:35.853981 20024 hybrid_clock.cc:648] HybridClock initialized: now 1790506715853962 us; error 30 us; skew 500 ppm
I20260927 10:58:35.855871 20024 webserver.cc:460] Webserver started at http://127.1.46.194:35681/ using document root <none> and password file <none>
I20260927 10:58:35.856482 20024 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:35.856564 20024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:35.856779 20024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:35.858808 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "4bd461ce80734b9abb4316e037746574"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "df45115dca92c4e6bd80f391cdd22ad1"
server_key_iv: "c8f366a8a2444a536d40553fd9a4cf06"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.859352 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "4bd461ce80734b9abb4316e037746574"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "df45115dca92c4e6bd80f391cdd22ad1"
server_key_iv: "c8f366a8a2444a536d40553fd9a4cf06"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.862835 20024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:58:35.865082 20038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:35.866295 20024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:35.866384 20024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "4bd461ce80734b9abb4316e037746574"
format_stamp: "Formatted at 2026-09-27 10:58:35 on dist-test-slave-cffh"
server_key: "df45115dca92c4e6bd80f391cdd22ad1"
server_key_iv: "c8f366a8a2444a536d40553fd9a4cf06"
server_key_version: "encryptionkey@0"
I20260927 10:58:35.866530 20024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:35.878053 20024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:35.878695 20024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:35.878866 20024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:35.879383 20024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:35.880230 20024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:35.880300 20024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:35.880337 20024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:35.880389 20024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:35.889279 20024 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:34899
I20260927 10:58:35.889298 20151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:34899 every 8 connection(s)
I20260927 10:58:35.890115 20024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:58:35.899267 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.891920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:51485 (local address 127.1.46.254:39915)
0927 10:58:35.892053 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:35.892056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:35.893107 (+ 1051us) server_negotiation.cc:407] Connection header received
0927 10:58:35.894553 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:35.894556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:35.894680 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:35.894755 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:35.895610 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.896142 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.897329 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.897541 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.897628 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.898794 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:35.898797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:35.898799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:35.898810 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:35.898830 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:35.898838 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:35.898902 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:35.899095 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 10:58:35.899156 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:58:35.899446 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20024
I20260927 10:58:35.899576 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:58:35.899901 1211 external_mini_cluster.cc:1428] Setting key f56f3b77e0b8eecc97aad9bbe7f800fb
I20260927 10:58:35.900249 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:35.892270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39915 (local address 127.1.46.194:51485)
0927 10:58:35.892970 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:35.893008 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:35.894367 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:35.894943 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:35.894949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:35.895025 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:35.895478 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.895490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.896275 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.896277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.897190 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:35.897198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:35.897708 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:35.897710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:35.897850 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:35.898385 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:35.898386 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:35.898422 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:35.898479 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:35.898923 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:35.898931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:35.898936 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:58:35.899036 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:58:35.899334 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":148,"threads_started":1}
I20260927 10:58:35.901341 20152 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39915
I20260927 10:58:35.901551 20152 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:35.901971 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39915
--builtin_ntp_servers=127.1.46.212:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:58:35.902043 20152 heartbeater.cc:511] Master 127.1.46.254:39915 requested a full tablet report, sending...
I20260927 10:58:35.903049 19833 ts_manager.cc:194] Registered new tserver with Master: 4bd461ce80734b9abb4316e037746574 (127.1.46.194:34899)
I20260927 10:58:35.903581 19833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.194:51485
W20260927 10:58:36.008728 20155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:36.008951 20155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:36.008987 20155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:36.012969 20155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:36.013015 20155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:36.013095 20155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:58:36.017797 20155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:39915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:36.018884 20155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:36.020231 20155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:36.029100 20163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:36.029125 20160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:36.029404 20155 server_base.cc:1034] running on GCE node
W20260927 10:58:36.029409 20161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:36.029842 20155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:36.030452 20155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:36.031626 20155 hybrid_clock.cc:648] HybridClock initialized: now 1790506716031606 us; error 36 us; skew 500 ppm
I20260927 10:58:36.033370 20155 webserver.cc:460] Webserver started at http://127.1.46.195:32857/ using document root <none> and password file <none>
I20260927 10:58:36.033903 20155 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:36.034003 20155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:36.034219 20155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:36.036104 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "4f02926714754c15afeae187c51f72cf"
format_stamp: "Formatted at 2026-09-27 10:58:36 on dist-test-slave-cffh"
server_key: "c495da3e181518a717dc209339c10ee0"
server_key_iv: "e8e3e6f7588a48ccb965bca37f3d057e"
server_key_version: "encryptionkey@0"
I20260927 10:58:36.036630 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "4f02926714754c15afeae187c51f72cf"
format_stamp: "Formatted at 2026-09-27 10:58:36 on dist-test-slave-cffh"
server_key: "c495da3e181518a717dc209339c10ee0"
server_key_iv: "e8e3e6f7588a48ccb965bca37f3d057e"
server_key_version: "encryptionkey@0"
I20260927 10:58:36.039986 20155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 10:58:36.042245 20169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:36.043459 20155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:58:36.043582 20155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "4f02926714754c15afeae187c51f72cf"
format_stamp: "Formatted at 2026-09-27 10:58:36 on dist-test-slave-cffh"
server_key: "c495da3e181518a717dc209339c10ee0"
server_key_iv: "e8e3e6f7588a48ccb965bca37f3d057e"
server_key_version: "encryptionkey@0"
I20260927 10:58:36.043695 20155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:36.063760 20155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:36.064352 20155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:36.064533 20155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:36.065050 20155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:36.065891 20155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:36.065958 20155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:36.066023 20155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:36.066071 20155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:36.074926 20155 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:40845
I20260927 10:58:36.074956 20282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:40845 every 8 connection(s)
I20260927 10:58:36.075753 20155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:58:36.077907 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20155
I20260927 10:58:36.078027 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:58:36.078385 1211 external_mini_cluster.cc:1428] Setting key eebff014323f328d3df60ab913eb24ca
I20260927 10:58:36.084743 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.077484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:56965 (local address 127.1.46.254:39915)
0927 10:58:36.077629 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.077634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.078885 (+ 1251us) server_negotiation.cc:407] Connection header received
0927 10:58:36.080459 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.080463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.080520 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.080587 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:36.081380 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.081893 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.083084 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.083315 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.083397 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.084355 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:36.084358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:36.084360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:36.084371 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:36.084389 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:36.084397 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:36.084456 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.084580 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.084628 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:58:36.085577 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.077878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:39915 (local address 127.1.46.195:56965)
0927 10:58:36.078716 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:36.078765 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:36.080247 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:36.080729 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:36.080735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:36.080813 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:36.081261 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.081272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.082000 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.082003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.082982 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.082992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.083437 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.083440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.083572 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.084127 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:36.084128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:36.084164 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:36.084226 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:36.084466 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:36.084473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:36.084476 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:58:36.084565 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:58:36.084808 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":174,"threads_started":1}
I20260927 10:58:36.086686 20283 heartbeater.cc:347] Connected to a master server at 127.1.46.254:39915
I20260927 10:58:36.086915 20283 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:36.087409 20283 heartbeater.cc:511] Master 127.1.46.254:39915 requested a full tablet report, sending...
I20260927 10:58:36.088371 19833 ts_manager.cc:194] Registered new tserver with Master: 4f02926714754c15afeae187c51f72cf (127.1.46.195:40845)
I20260927 10:58:36.088928 19833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.195:56965
I20260927 10:58:36.092458 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:58:36.098557 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.094839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.1.46.254:39915)
0927 10:58:36.094985 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.094990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.095216 (+ 226us) server_negotiation.cc:407] Connection header received
0927 10:58:36.095278 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.095280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.095319 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.095389 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:36.095594 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.096119 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.097000 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.097209 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.097292 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.098062 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:36.098065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:36.098066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:36.098077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:36.098094 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:36.098101 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:36.098201 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.098321 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.098391 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:58:36.101596 19833 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:58:36.104748 19833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:58:36.119671 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.113426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:34707 (local address 127.0.0.1:33304)
0927 10:58:36.115054 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:36.115067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:36.115233 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:36.115627 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:36.115631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:36.115647 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:36.115726 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.115731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.116741 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.116744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.118249 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.118260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.119117 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.119121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.119225 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.119259 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:58:36.119373 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 10:58:36.119488 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1573,"mutex_wait_us":101,"thread_start_us":80,"threads_started":1}
I20260927 10:58:36.119669 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.113882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.1.46.193:34707)
0927 10:58:36.114281 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.114295 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.115165 (+ 870us) server_negotiation.cc:407] Connection header received
0927 10:58:36.115369 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.115373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.115499 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.115656 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:36.115821 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.116580 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.118400 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.118998 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.119120 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.119321 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.119407 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.119499 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260927 10:58:36.119858 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.113148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:34899 (local address 127.0.0.1:57452)
0927 10:58:36.114170 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:36.114184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:36.114266 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:36.114874 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:36.114880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:36.114907 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:36.115051 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.115059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.116557 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.116561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.118117 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.118127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.119477 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.119480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.119570 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.119590 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:58:36.119671 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:58:36.119729 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":100,"threads_started":1}
I20260927 10:58:36.119714 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.113088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:40845 (local address 127.0.0.1:39848)
0927 10:58:36.113724 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:36.113772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:36.113906 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:36.115006 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:36.115010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:36.115028 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:36.115154 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.115161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.116582 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.116585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.118249 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.118260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.119281 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.119284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.119384 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.119406 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:58:36.119461 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 10:58:36.119528 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":187,"thread_start_us":104,"threads_started":1}
I20260927 10:58:36.120292 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.113882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.1.46.194:34899)
0927 10:58:36.114345 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.114356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.114384 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:58:36.114474 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.114478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.114712 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.114837 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:36.115181 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.116441 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.118260 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.119096 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.119216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.119418 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.120033 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.120133 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":140,"threads_started":1}
I20260927 10:58:36.120564 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.113225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.1.46.195:40845)
0927 10:58:36.114480 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.114491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.114517 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:58:36.114618 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.114624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.114846 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.115005 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:36.115258 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.116433 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.118388 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.119150 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.119814 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.120268 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.120321 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.120413 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":122,"threads_started":1}
I20260927 10:58:36.122843 20086 tablet_service.cc:1467] Processing CreateTablet for tablet 2122783da67148759305260039037335 (DEFAULT_TABLE table=test-table [id=9e5502ebfc2b4fae9561562d7270fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:36.122937 19955 tablet_service.cc:1467] Processing CreateTablet for tablet 2122783da67148759305260039037335 (DEFAULT_TABLE table=test-table [id=9e5502ebfc2b4fae9561562d7270fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:36.122937 20217 tablet_service.cc:1467] Processing CreateTablet for tablet 2122783da67148759305260039037335 (DEFAULT_TABLE table=test-table [id=9e5502ebfc2b4fae9561562d7270fcd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:36.123838 20086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2122783da67148759305260039037335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:36.123906 19955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2122783da67148759305260039037335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:36.123906 20217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2122783da67148759305260039037335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:36.130300 20300 tablet_bootstrap.cc:492] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Bootstrap starting.
I20260927 10:58:36.131124 20301 tablet_bootstrap.cc:492] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: Bootstrap starting.
I20260927 10:58:36.132606 20300 tablet_bootstrap.cc:654] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:36.133235 20301 tablet_bootstrap.cc:654] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:36.133499 20300 log.cc:824] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:36.134049 20301 log.cc:824] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:36.134218 20302 tablet_bootstrap.cc:492] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Bootstrap starting.
I20260927 10:58:36.135582 20300 tablet_bootstrap.cc:492] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: No bootstrap required, opened a new log
I20260927 10:58:36.135787 20300 ts_tablet_manager.cc:1397] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:58:36.135898 20301 tablet_bootstrap.cc:492] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: No bootstrap required, opened a new log
I20260927 10:58:36.136080 20301 ts_tablet_manager.cc:1397] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:58:36.136363 20302 tablet_bootstrap.cc:654] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:36.136988 20302 log.cc:824] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:36.138446 20302 tablet_bootstrap.cc:492] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: No bootstrap required, opened a new log
I20260927 10:58:36.138652 20302 ts_tablet_manager.cc:1397] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:58:36.140048 20300 raft_consensus.cc:348] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.140048 20301 raft_consensus.cc:348] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.140296 20301 raft_consensus.cc:374] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:36.140296 20300 raft_consensus.cc:374] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:36.140362 20300 raft_consensus.cc:729] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd461ce80734b9abb4316e037746574, State: Initialized, Role: FOLLOWER
I20260927 10:58:36.140383 20301 raft_consensus.cc:729] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f627a1c5594d4062960460739254017f, State: Initialized, Role: FOLLOWER
I20260927 10:58:36.140826 20300 consensus_queue.cc:260] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.140823 20301 consensus_queue.cc:260] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.141629 20152 heartbeater.cc:503] Master 127.1.46.254:39915 was elected leader, sending a full tablet report...
I20260927 10:58:36.141659 20021 heartbeater.cc:503] Master 127.1.46.254:39915 was elected leader, sending a full tablet report...
I20260927 10:58:36.141758 20300 ts_tablet_manager.cc:1428] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:58:36.141758 20301 ts_tablet_manager.cc:1428] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260927 10:58:36.143239 20302 raft_consensus.cc:348] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.143493 20302 raft_consensus.cc:374] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:36.143556 20302 raft_consensus.cc:729] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f02926714754c15afeae187c51f72cf, State: Initialized, Role: FOLLOWER
I20260927 10:58:36.143946 20302 consensus_queue.cc:260] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.144585 20283 heartbeater.cc:503] Master 127.1.46.254:39915 was elected leader, sending a full tablet report...
I20260927 10:58:36.144963 20302 ts_tablet_manager.cc:1428] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 10:58:36.215811 20022 tablet.cc:2369] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:36.326756 20284 tablet.cc:2369] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:36.391247 20153 tablet.cc:2369] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:36.438879 20306 raft_consensus.cc:482] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:36.439145 20306 raft_consensus.cc:504] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.440347 20306 leader_election.cc:288] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f02926714754c15afeae187c51f72cf (127.1.46.195:40845), f627a1c5594d4062960460739254017f (127.1.46.193:34707)
I20260927 10:58:36.445684 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.440852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:44473 (local address 127.1.46.193:34707)
0927 10:58:36.441010 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.441016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.441029 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:58:36.441099 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.441102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.441150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.441228 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:36.441629 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.442391 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.444240 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.444903 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.445030 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.445081 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.445487 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.445551 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:58:36.445703 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.440666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:44325 (local address 127.1.46.195:40845)
0927 10:58:36.440801 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.440807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.440903 (+ 96us) server_negotiation.cc:407] Connection header received
0927 10:58:36.441061 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.441067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.441119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.441203 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:36.441629 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.442390 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.444240 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.444903 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.445030 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.445081 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.445487 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.445551 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:58:36.445762 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.440529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:40845 (local address 127.1.46.194:44325)
0927 10:58:36.440786 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:36.440803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:36.440922 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:36.441242 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:36.441245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:36.441276 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:36.441511 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.441516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.442510 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.442514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.444106 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.444118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.445069 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.445074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.445293 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.445363 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 10:58:36.445463 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:58:36.445569 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":359,"spinlock_wait_cycles":9344,"thread_start_us":101,"threads_started":1}
I20260927 10:58:36.445819 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.440633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:34707 (local address 127.1.46.194:44473)
0927 10:58:36.440933 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:36.440944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:36.441010 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:36.441242 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:36.441245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:36.441271 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:36.441506 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.441514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.442510 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.442514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.444122 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:36.444128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.445069 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.445074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:36.445271 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.445359 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 10:58:36.445463 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:58:36.445569 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":270,"spinlock_wait_cycles":768,"thread_start_us":61,"threads_started":1}
I20260927 10:58:36.446600 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2122783da67148759305260039037335" candidate_uuid: "4bd461ce80734b9abb4316e037746574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f627a1c5594d4062960460739254017f" is_pre_election: true
I20260927 10:58:36.446758 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2122783da67148759305260039037335" candidate_uuid: "4bd461ce80734b9abb4316e037746574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f02926714754c15afeae187c51f72cf" is_pre_election: true
I20260927 10:58:36.446978 19975 raft_consensus.cc:2454] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bd461ce80734b9abb4316e037746574 in term 0.
I20260927 10:58:36.447046 20237 raft_consensus.cc:2454] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bd461ce80734b9abb4316e037746574 in term 0.
I20260927 10:58:36.447567 20042 leader_election.cc:302] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd461ce80734b9abb4316e037746574, f627a1c5594d4062960460739254017f; no voters:
I20260927 10:58:36.447873 20306 raft_consensus.cc:2781] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:58:36.447974 20306 raft_consensus.cc:482] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:58:36.448040 20306 raft_consensus.cc:3037] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:36.449235 20306 raft_consensus.cc:504] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.449640 20306 leader_election.cc:288] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [CANDIDATE]: Term 1 election: Requested vote from peers 4f02926714754c15afeae187c51f72cf (127.1.46.195:40845), f627a1c5594d4062960460739254017f (127.1.46.193:34707)
I20260927 10:58:36.449904 20237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2122783da67148759305260039037335" candidate_uuid: "4bd461ce80734b9abb4316e037746574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f02926714754c15afeae187c51f72cf"
I20260927 10:58:36.450017 19975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2122783da67148759305260039037335" candidate_uuid: "4bd461ce80734b9abb4316e037746574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f627a1c5594d4062960460739254017f"
I20260927 10:58:36.450040 20237 raft_consensus.cc:3037] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:36.450132 19975 raft_consensus.cc:3037] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:36.451326 20237 raft_consensus.cc:2454] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bd461ce80734b9abb4316e037746574 in term 1.
I20260927 10:58:36.451326 19975 raft_consensus.cc:2454] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bd461ce80734b9abb4316e037746574 in term 1.
I20260927 10:58:36.451640 20040 leader_election.cc:302] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4bd461ce80734b9abb4316e037746574, 4f02926714754c15afeae187c51f72cf, f627a1c5594d4062960460739254017f; no voters:
I20260927 10:58:36.451855 20306 raft_consensus.cc:2781] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:36.452072 20306 raft_consensus.cc:686] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd461ce80734b9abb4316e037746574, State: Running, Role: LEADER
I20260927 10:58:36.452433 20306 consensus_queue.cc:237] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } }
I20260927 10:58:36.455526 19832 catalog_manager.cc:5495] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd461ce80734b9abb4316e037746574 (127.1.46.194). New cstate: current_term: 1 leader_uuid: "4bd461ce80734b9abb4316e037746574" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bd461ce80734b9abb4316e037746574" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 } health_report { overall_health: UNKNOWN } } }
I20260927 10:58:36.483028 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.479086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.1.46.194:34899)
0927 10:58:36.479208 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:36.479212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:36.479226 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:58:36.479381 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:36.479383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:36.479438 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:36.479530 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:58:36.479772 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.480257 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.481479 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:36.481737 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:36.481824 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:36.482145 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:58:36.482499 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:58:36.482758 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:36.482841 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 10:58:36.482871 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:58:36.488687 20237 raft_consensus.cc:1261] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 1 FOLLOWER]: Refusing update from remote peer 4bd461ce80734b9abb4316e037746574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:36.488683 19975 raft_consensus.cc:1261] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 1 FOLLOWER]: Refusing update from remote peer 4bd461ce80734b9abb4316e037746574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:36.489460 20311 consensus_queue.cc:1035] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f627a1c5594d4062960460739254017f" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:36.489687 20306 consensus_queue.cc:1035] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f02926714754c15afeae187c51f72cf" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:36.496058 20314 mvcc.cc:205] Tried to move back new op lower bound from 7333915510729859072 to 7333915510591942656. Current Snapshot: MvccSnapshot[applied={T|T < 7333915510729859072}]
I20260927 10:58:36.499436 20315 mvcc.cc:205] Tried to move back new op lower bound from 7333915510729859072 to 7333915510591942656. Current Snapshot: MvccSnapshot[applied={T|T < 7333915510729859072}]
I20260927 10:58:36.499722 20316 mvcc.cc:205] Tried to move back new op lower bound from 7333915510729859072 to 7333915510591942656. Current Snapshot: MvccSnapshot[applied={T|T < 7333915510729859072}]
I20260927 10:58:36.503971 19832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:36.504132 19832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:36.506458 19832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06ca98296b6648fba4f68f218093ba75: Sending DeleteTablet for 3 replicas of tablet 2122783da67148759305260039037335
I20260927 10:58:36.507190 20086 tablet_service.cc:1514] Processing DeleteTablet for tablet 2122783da67148759305260039037335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:36 UTC) from {username='slave'} at 127.0.0.1:57452
I20260927 10:58:36.507174 20217 tablet_service.cc:1514] Processing DeleteTablet for tablet 2122783da67148759305260039037335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:36 UTC) from {username='slave'} at 127.0.0.1:39848
I20260927 10:58:36.507591 20324 tablet_replica.cc:330] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: stopping tablet replica
I20260927 10:58:36.507625 19955 tablet_service.cc:1514] Processing DeleteTablet for tablet 2122783da67148759305260039037335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:36 UTC) from {username='slave'} at 127.0.0.1:33304
I20260927 10:58:36.507951 20324 raft_consensus.cc:2229] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:36.508075 20325 tablet_replica.cc:330] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f: stopping tablet replica
I20260927 10:58:36.508219 20324 raft_consensus.cc:2258] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:36.508356 20325 raft_consensus.cc:2229] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:36.508667 20325 raft_consensus.cc:2258] T 2122783da67148759305260039037335 P f627a1c5594d4062960460739254017f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:36.508767 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 19893
I20260927 10:58:36.509716 20324 ts_tablet_manager.cc:1905] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:36.511641 20323 tablet_replica.cc:330] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: stopping tablet replica
I20260927 10:58:36.511942 20323 raft_consensus.cc:2229] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:58:36.512305 20323 raft_consensus.cc:2258] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:36.513243 20324 ts_tablet_manager.cc:1918] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:36.513387 20324 log.cc:1196] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/2122783da67148759305260039037335
I20260927 10:58:36.513518 20323 ts_tablet_manager.cc:1905] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:36.513756 20324 ts_tablet_manager.cc:1939] T 2122783da67148759305260039037335 P 4f02926714754c15afeae187c51f72cf: Deleting consensus metadata
I20260927 10:58:36.515120 19818 catalog_manager.cc:4841] TS 4f02926714754c15afeae187c51f72cf (127.1.46.195:40845): tablet 2122783da67148759305260039037335 (table test-table [id=9e5502ebfc2b4fae9561562d7270fcd0]) successfully deleted
I20260927 10:58:36.516263 20323 ts_tablet_manager.cc:1918] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:36.516376 20323 log.cc:1196] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/2122783da67148759305260039037335
I20260927 10:58:36.516676 20323 ts_tablet_manager.cc:1939] T 2122783da67148759305260039037335 P 4bd461ce80734b9abb4316e037746574: Deleting consensus metadata
I20260927 10:58:36.517773 19817 catalog_manager.cc:4841] TS 4bd461ce80734b9abb4316e037746574 (127.1.46.194:34899): tablet 2122783da67148759305260039037335 (table test-table [id=9e5502ebfc2b4fae9561562d7270fcd0]) successfully deleted
W20260927 10:58:36.518986 19820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:34707 (error 108)
I20260927 10:58:36.519099 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20024
I20260927 10:58:36.520545 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:36.520179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:34707 (local address 127.0.0.1:33310)
0927 10:58:36.520372 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:36.520448 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 10:58:36.520705 19820 catalog_manager.cc:4584] TS f627a1c5594d4062960460739254017f (127.1.46.193:34707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2122783da67148759305260039037335: Network error: Client connection negotiation failed: client connection to 127.1.46.193:34707: connect: Connection refused (error 111)
I20260927 10:58:36.527369 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20155
I20260927 10:58:36.532924 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 19802
2026-09-27T10:58:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1259 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:36 dist-test-slave-cffh krb5kdc[20329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:36 dist-test-slave-cffh krb5kdc[20329](info): set up 2 sockets
Sep 27 10:58:36 dist-test-slave-cffh krb5kdc[20329](info): commencing operation
krb5kdc: starting...
W20260927 10:58:37.554303 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:37 dist-test-slave-cffh krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:58:37.671837 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45835
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:45835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:37.778085 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:37.778389 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:37.778463 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:37.782363 20342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:37.782447 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:37.782487 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:37.782521 20342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:37.782555 20342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:37.787586 20342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:45835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:37.788527 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:37.789501 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:37.797539 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:37.797686 20342 server_base.cc:1034] running on GCE node
W20260927 10:58:37.797526 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:37.797894 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:37.798419 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:37.799506 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:37.800679 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790506717800658 us; error 31 us; skew 500 ppm
Sep 27 10:58:37 dist-test-slave-cffh krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506717, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:37.803483 20342 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:58:37.804391 20342 webserver.cc:460] Webserver started at http://127.1.46.254:41355/ using document root <none> and password file <none>
I20260927 10:58:37.804874 20342 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:37.804972 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:37.805212 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:37.807225 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "3902b2b38364410c88965ce838a75974"
format_stamp: "Formatted at 2026-09-27 10:58:37 on dist-test-slave-cffh"
server_key: "c41022ebc4728d84f0a74eb64d5b5b0f"
server_key_iv: "2cdd08632ddaad94888c0e5246696831"
server_key_version: "encryptionkey@0"
I20260927 10:58:37.807821 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "3902b2b38364410c88965ce838a75974"
format_stamp: "Formatted at 2026-09-27 10:58:37 on dist-test-slave-cffh"
server_key: "c41022ebc4728d84f0a74eb64d5b5b0f"
server_key_iv: "2cdd08632ddaad94888c0e5246696831"
server_key_version: "encryptionkey@0"
I20260927 10:58:37.811396 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:37.813724 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:37.815091 20342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:37.815235 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "3902b2b38364410c88965ce838a75974"
format_stamp: "Formatted at 2026-09-27 10:58:37 on dist-test-slave-cffh"
server_key: "c41022ebc4728d84f0a74eb64d5b5b0f"
server_key_iv: "2cdd08632ddaad94888c0e5246696831"
server_key_version: "encryptionkey@0"
I20260927 10:58:37.815348 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:37.840965 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:37.843626 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:37.843839 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:37.851022 20342 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:45835
I20260927 10:58:37.851059 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:45835 every 8 connection(s)
I20260927 10:58:37.851958 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:37.854830 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:37.857900 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20342
I20260927 10:58:37.858059 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:37.858454 1211 external_mini_cluster.cc:1428] Setting key ee3a08c1ee58a7aeda8d649c67717125
I20260927 10:58:37.861021 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974: Bootstrap starting.
I20260927 10:58:37.863162 20410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:37.863858 20410 log.cc:824] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:37.865612 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974: No bootstrap required, opened a new log
Sep 27 10:58:37 dist-test-slave-cffh krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:37.869354 20410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3902b2b38364410c88965ce838a75974" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45835 } }
I20260927 10:58:37.869586 20410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:37.869648 20410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3902b2b38364410c88965ce838a75974, State: Initialized, Role: FOLLOWER
I20260927 10:58:37.870110 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3902b2b38364410c88965ce838a75974" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45835 } }
I20260927 10:58:37.870255 20410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:37.870342 20410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:37.870438 20410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:37.871328 20410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3902b2b38364410c88965ce838a75974" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45835 } }
I20260927 10:58:37.871665 20410 leader_election.cc:302] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3902b2b38364410c88965ce838a75974; no voters:
I20260927 10:58:37.871946 20410 leader_election.cc:288] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:37.872051 20415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:37.872232 20415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [term 1 LEADER]: Becoming Leader. State: Replica: 3902b2b38364410c88965ce838a75974, State: Running, Role: LEADER
I20260927 10:58:37.872601 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:37.859918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.1.46.254:45835)
0927 10:58:37.860341 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:37.860355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:37.860388 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:58:37.861115 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:37.861139 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:37.861455 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:37.861762 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:37.862619 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:37.863633 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:37.864728 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:37.865184 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:37.865358 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:37.867859 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:37.867864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:37.867876 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:37.867900 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:37.869493 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:37.869992 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:37.869997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:37.870001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:37.870105 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:37.870406 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:37.870410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:37.870413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:37.870867 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:37.871069 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:37.871337 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 10:58:37.871638 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260927 10:58:37.872565 20415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3902b2b38364410c88965ce838a75974" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45835 } }
I20260927 10:58:37.873236 20410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:37.873870 20417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3902b2b38364410c88965ce838a75974. Latest consensus state: current_term: 1 leader_uuid: "3902b2b38364410c88965ce838a75974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3902b2b38364410c88965ce838a75974" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45835 } } }
I20260927 10:58:37.873978 20417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:37.875280 20423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:37.875636 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3902b2b38364410c88965ce838a75974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3902b2b38364410c88965ce838a75974" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 45835 } } }
I20260927 10:58:37.875744 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:37.877728 20423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:37.882527 20423 catalog_manager.cc:1347] Generated new cluster ID: 2949ae6ed9504c039b462542d3bd160c
I20260927 10:58:37.882608 20423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:37.904028 20423 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:37.905193 20423 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:37.916114 20423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974: Generated new TSK 0
I20260927 10:58:37.916782 20423 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
Entry for principal kudu/127.1.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab.
I20260927 10:58:37.945619 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45835
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:38.061815 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:38.062039 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:38.062078 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:38.065968 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:38.066016 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:38.066069 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:58:38.070959 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.193.keytab
--principal=kudu/127.1.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:38.071805 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:38.072782 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:38.081912 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:38.081918 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:38.081933 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:38.082419 20436 server_base.cc:1034] running on GCE node
I20260927 10:58:38.082796 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:38.083338 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:38.084522 20436 hybrid_clock.cc:648] HybridClock initialized: now 1790506718084481 us; error 48 us; skew 500 ppm
Sep 27 10:58:38 dist-test-slave-cffh krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:38.087528 20436 init.cc:377] Logged in from keytab as kudu/127.1.46.193@KRBTEST.COM (short username kudu)
I20260927 10:58:38.088399 20436 webserver.cc:460] Webserver started at http://127.1.46.193:45045/ using document root <none> and password file <none>
I20260927 10:58:38.088939 20436 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:38.089030 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:38.089264 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:38.091349 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "08e167c6a4e5464e88cc8f8fe1239f20"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "f6203b3dbc9f67453b6279fd1b43ff03"
server_key_iv: "2c29e5848a19ad814f16ea0cd361f736"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.091898 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "08e167c6a4e5464e88cc8f8fe1239f20"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "f6203b3dbc9f67453b6279fd1b43ff03"
server_key_iv: "2c29e5848a19ad814f16ea0cd361f736"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.095515 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:58:38.097781 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.098989 20436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:58:38.099113 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "08e167c6a4e5464e88cc8f8fe1239f20"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "f6203b3dbc9f67453b6279fd1b43ff03"
server_key_iv: "2c29e5848a19ad814f16ea0cd361f736"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.099231 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:38.116549 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:38.119665 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:38.119848 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:38.120366 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:38.121176 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:38.121241 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.121305 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:38.121358 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.129882 20436 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:44337
I20260927 10:58:38.129944 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:44337 every 8 connection(s)
I20260927 10:58:38.130728 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:58:38.131398 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20436
I20260927 10:58:38.131559 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:58:38.131844 1211 external_mini_cluster.cc:1428] Setting key dc0a111796b54d6f114853d73169d529
Sep 27 10:58:38 dist-test-slave-cffh krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.193@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.194@KRBTEST.COM" created.
I20260927 10:58:38.143920 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.134414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:40459 (local address 127.1.46.254:45835)
0927 10:58:38.134586 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.134593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.134610 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:58:38.134653 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.134659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.134715 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.134795 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:38.135871 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.136400 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.137499 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.137755 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.137864 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.140200 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:38.140204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:38.140208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:38.140238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:38.141832 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.142363 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.142368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.142371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.142459 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.142807 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.142813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.142815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.143062 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:38.143170 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.143634 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.143752 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:58:38.144801 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.132859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45835 (local address 127.1.46.193:40459)
0927 10:58:38.133380 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.133418 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.134096 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.134960 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.134966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.135302 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:38.135730 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.135742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.136522 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.136526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.137366 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.137375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.137895 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.137899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.138061 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.138676 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:38.138691 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:38.140041 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:38.141965 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:38.141971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:38.141984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:38.142262 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:38.142536 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:38.142541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:38.142544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:38.142711 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:38.143170 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:38.143176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:38.143404 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.143583 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.143871 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":154,"threads_started":1}
I20260927 10:58:38.146003 20565 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45835
I20260927 10:58:38.146235 20565 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:38.146813 20565 heartbeater.cc:511] Master 127.1.46.254:45835 requested a full tablet report, sending...
I20260927 10:58:38.148564 20374 ts_manager.cc:194] Registered new tserver with Master: 08e167c6a4e5464e88cc8f8fe1239f20 (127.1.46.193:44337)
I20260927 10:58:38.150199 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.193@KRBTEST.COM'} at 127.1.46.193:40459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
Entry for principal kudu/127.1.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab.
I20260927 10:58:38.158690 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45835
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:38.265015 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:38.265247 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:38.265287 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:38.269213 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:38.269263 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:38.269359 20570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:58:38.274428 20570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.194.keytab
--principal=kudu/127.1.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:38.275429 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:38.276381 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:58:38.285629 20570 server_base.cc:1034] running on GCE node
W20260927 10:58:38.285576 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:38.285580 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:38.285601 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:38.286219 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:38.286971 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:38.288126 20570 hybrid_clock.cc:648] HybridClock initialized: now 1790506718288098 us; error 43 us; skew 500 ppm
Sep 27 10:58:38 dist-test-slave-cffh krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:38.290866 20570 init.cc:377] Logged in from keytab as kudu/127.1.46.194@KRBTEST.COM (short username kudu)
I20260927 10:58:38.291651 20570 webserver.cc:460] Webserver started at http://127.1.46.194:38551/ using document root <none> and password file <none>
I20260927 10:58:38.292183 20570 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:38.292243 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:38.292481 20570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:38.294430 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "ae581ac897604d99a07c26fa2b7fba7e"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "e8741ee5bb60cfbeb2b24605b64f782d"
server_key_iv: "3ee4b1ccd03080bc4a823ccf71c07dd8"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.295030 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "ae581ac897604d99a07c26fa2b7fba7e"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "e8741ee5bb60cfbeb2b24605b64f782d"
server_key_iv: "3ee4b1ccd03080bc4a823ccf71c07dd8"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.298357 20570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:38.300675 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.301819 20570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:38.301944 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "ae581ac897604d99a07c26fa2b7fba7e"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "e8741ee5bb60cfbeb2b24605b64f782d"
server_key_iv: "3ee4b1ccd03080bc4a823ccf71c07dd8"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.302047 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:38.323061 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:38.325740 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:38.325935 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:38.326470 20570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:38.327261 20570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:38.327330 20570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.327414 20570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:38.327476 20570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.336613 20570 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:42709
I20260927 10:58:38.336640 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:42709 every 8 connection(s)
I20260927 10:58:38.337417 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:58:38.344244 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20570
I20260927 10:58:38.344359 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:58:38.344642 1211 external_mini_cluster.cc:1428] Setting key c25e34cf914ae59498986c2f9c655207
Sep 27 10:58:38 dist-test-slave-cffh krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.194@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:38.350472 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.339226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:49601 (local address 127.1.46.254:45835)
0927 10:58:38.339352 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.339356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.340090 (+ 734us) server_negotiation.cc:407] Connection header received
0927 10:58:38.340861 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.340864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.340910 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.340981 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:38.342080 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.342603 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.343604 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.343817 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.343888 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.346485 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:38.346488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:38.346491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:38.346516 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:38.348181 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.348821 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.348823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.348824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.348896 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.349284 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.349287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.349288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.349547 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:38.349648 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.350081 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.350228 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:58:38.351022 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.339509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45835 (local address 127.1.46.194:49601)
0927 10:58:38.339941 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.339978 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.340680 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.341154 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.341160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.341474 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:38.341948 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.341961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.342697 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.342700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.343508 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.343516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.343930 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.343932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.344066 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.344662 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:38.344676 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:38.346332 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:38.348311 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:38.348319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:38.348335 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:38.348722 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:38.348987 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:38.348993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:38.348997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:38.349181 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:38.349647 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:38.349652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:38.349856 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.350002 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.350225 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260927 10:58:38.352098 20699 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45835
I20260927 10:58:38.352324 20699 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:38.352766 20699 heartbeater.cc:511] Master 127.1.46.254:45835 requested a full tablet report, sending...
I20260927 10:58:38.353708 20374 ts_manager.cc:194] Registered new tserver with Master: ae581ac897604d99a07c26fa2b7fba7e (127.1.46.194:42709)
I20260927 10:58:38.354540 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.194@KRBTEST.COM'} at 127.1.46.194:49601
WARNING: no policy specified for kudu/127.1.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
Entry for principal kudu/127.1.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab.
I20260927 10:58:38.372860 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45835
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:38.477918 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:38.478142 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:38.478189 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:38.482048 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:38.482093 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:38.482146 20704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:58:38.487078 20704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.195.keytab
--principal=kudu/127.1.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:45835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:38.487987 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:38.488895 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:38.497812 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:38.497821 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:38.497841 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:38.497941 20704 server_base.cc:1034] running on GCE node
I20260927 10:58:38.498690 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:38.499282 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:38.500458 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790506718500429 us; error 49 us; skew 500 ppm
Sep 27 10:58:38 dist-test-slave-cffh krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:38.503057 20704 init.cc:377] Logged in from keytab as kudu/127.1.46.195@KRBTEST.COM (short username kudu)
I20260927 10:58:38.503903 20704 webserver.cc:460] Webserver started at http://127.1.46.195:37379/ using document root <none> and password file <none>
I20260927 10:58:38.504420 20704 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:38.504508 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:38.504768 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:38.506879 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "21e12e5ca8044904a998f3926638ddef"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "9715bf60e64ff767401270fdab6271dc"
server_key_iv: "3eed304d11eff07830544e01f93112f7"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.507416 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "21e12e5ca8044904a998f3926638ddef"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "9715bf60e64ff767401270fdab6271dc"
server_key_iv: "3eed304d11eff07830544e01f93112f7"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.510841 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:38.513042 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.514127 20704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:38.514273 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "21e12e5ca8044904a998f3926638ddef"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "9715bf60e64ff767401270fdab6271dc"
server_key_iv: "3eed304d11eff07830544e01f93112f7"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.514377 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:38.536713 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:38.539373 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:38.539559 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:38.540117 20704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:38.540872 20704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:38.540942 20704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.541006 20704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:38.541054 20704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.549304 20704 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:38145
I20260927 10:58:38.549335 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:38145 every 8 connection(s)
I20260927 10:58:38.550100 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:58:38 dist-test-slave-cffh krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506718, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.195@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:38.558422 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20704
I20260927 10:58:38.558554 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:58:38.558923 1211 external_mini_cluster.cc:1428] Setting key bd3f954acc65dd4d6a385ad781485bf6
I20260927 10:58:38.562309 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.551685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:51505 (local address 127.1.46.254:45835)
0927 10:58:38.551820 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.551823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.552533 (+ 710us) server_negotiation.cc:407] Connection header received
0927 10:58:38.553318 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.553321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.553369 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.553440 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:38.554444 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.555020 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.556074 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.556280 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.556355 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.558624 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:38.558627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:38.558629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:38.558688 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:38.560339 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.560887 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.560893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.560898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.560989 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.561321 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.561326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.561329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.561493 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:38.561598 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.562097 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.562198 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:58:38.563387 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.551988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:45835 (local address 127.1.46.195:51505)
0927 10:58:38.552392 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.552428 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.553136 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.553588 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.553594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.553910 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:38.554328 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.554340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.555120 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.555123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.555966 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.555974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.556392 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.556395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.556538 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.556999 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:58:38.557013 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:38.558440 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:38.560436 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:38.560441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:38.560455 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:38.560777 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:38.561085 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:58:38.561088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:58:38.561091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:58:38.561225 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:58:38.561594 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:38.561600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:38.561821 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.562454 (+ 633us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.562761 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260927 10:58:38.564350 20833 heartbeater.cc:347] Connected to a master server at 127.1.46.254:45835
I20260927 10:58:38.564553 20833 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:38.564963 20833 heartbeater.cc:511] Master 127.1.46.254:45835 requested a full tablet report, sending...
I20260927 10:58:38.565840 20374 ts_manager.cc:194] Registered new tserver with Master: 21e12e5ca8044904a998f3926638ddef (127.1.46.195:38145)
I20260927 10:58:38.566408 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.46.195@KRBTEST.COM'} at 127.1.46.195:51505
I20260927 10:58:38.572500 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:58:38.580778 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.574359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.1.46.254:45835)
0927 10:58:38.574490 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.574495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.574751 (+ 256us) server_negotiation.cc:407] Connection header received
0927 10:58:38.574819 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.574822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.574855 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.574913 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:38.575510 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.575972 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.576704 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.576906 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.576978 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.578133 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:38.578136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:38.578137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:38.578157 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:38.579394 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.579765 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.579768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.579769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.579837 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:38.580047 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:38.580049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:38.580050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:38.580226 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:38.580324 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.580490 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.580570 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:58:38.583153 20374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:58:38.585887 20374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:58:38.599889 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.593714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:38145 (local address 127.0.0.1:52194)
0927 10:58:38.594058 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.594111 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.594225 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.595248 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.595254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.595280 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:38.595602 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.595612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.597063 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.597066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.598408 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.598417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.599409 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.599412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.599502 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.599530 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.599628 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.599749 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260927 10:58:38.599958 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.594414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.1.46.195:38145)
0927 10:58:38.594747 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.594759 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.594785 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:58:38.594865 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.594873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.595084 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.595242 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:38.595771 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.596924 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.598525 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.599299 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.599406 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.599552 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.599657 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.599777 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":143,"threads_started":1}
I20260927 10:58:38.600360 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.593924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:42709 (local address 127.0.0.1:33552)
0927 10:58:38.594828 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.594841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.594919 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.596143 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.596147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.596167 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:38.596265 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.596272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.597594 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.597597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.598813 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.598820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.599917 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.599923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.600027 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.600061 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.600132 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.600211 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":38,"thread_start_us":80,"threads_started":1}
I20260927 10:58:38.600792 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.595107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.1.46.194:42709)
0927 10:58:38.595510 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.595518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.595559 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:58:38.595635 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.595640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.595815 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.595966 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:38.596391 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.597428 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.598929 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.599779 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.599888 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.600042 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.600550 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.600644 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":130,"threads_started":1}
I20260927 10:58:38.603246 20767 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a (DEFAULT_TABLE table=test-table [id=6efb3852a60c46769c83ae86d0e751d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:38.603267 20633 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a (DEFAULT_TABLE table=test-table [id=6efb3852a60c46769c83ae86d0e751d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:38.604133 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:38.604144 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:38.610308 20851 tablet_bootstrap.cc:492] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Bootstrap starting.
I20260927 10:58:38.610450 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.594429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:44337 (local address 127.0.0.1:54166)
0927 10:58:38.595470 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.595484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.595581 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.595958 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.595961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.595980 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:38.596070 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.596077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.597233 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.597236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.598800 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.598810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.609920 (+ 11110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.609923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.610092 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.610117 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.610190 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.610265 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":49,"thread_start_us":101,"threads_started":1}
I20260927 10:58:38.610718 20850 tablet_bootstrap.cc:492] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Bootstrap starting.
I20260927 10:58:38.610960 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.594663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.1.46.193:44337)
0927 10:58:38.595061 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.595074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.595589 (+ 515us) server_negotiation.cc:407] Connection header received
0927 10:58:38.595667 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.595671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.595815 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.595966 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:38.596169 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.597086 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.600106 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.609570 (+ 9464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.609692 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.609855 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.610675 (+ 820us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.610818 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260927 10:58:38.612884 20850 tablet_bootstrap.cc:654] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:38.614082 20850 log.cc:824] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:38.613744 20499 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a (DEFAULT_TABLE table=test-table [id=6efb3852a60c46769c83ae86d0e751d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:38.614744 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:38.615146 20851 tablet_bootstrap.cc:654] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:38.615967 20851 log.cc:824] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:38.616093 20850 tablet_bootstrap.cc:492] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: No bootstrap required, opened a new log
I20260927 10:58:38.616310 20850 ts_tablet_manager.cc:1397] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:58:38.618036 20851 tablet_bootstrap.cc:492] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: No bootstrap required, opened a new log
I20260927 10:58:38.618206 20851 ts_tablet_manager.cc:1397] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 10:58:38.619848 20850 raft_consensus.cc:348] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.620175 20850 raft_consensus.cc:374] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:38.620265 20850 raft_consensus.cc:729] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e12e5ca8044904a998f3926638ddef, State: Initialized, Role: FOLLOWER
I20260927 10:58:38.620697 20850 consensus_queue.cc:260] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.621266 20833 heartbeater.cc:503] Master 127.1.46.254:45835 was elected leader, sending a full tablet report...
I20260927 10:58:38.621595 20850 ts_tablet_manager.cc:1428] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:58:38.622151 20855 tablet_bootstrap.cc:492] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20: Bootstrap starting.
I20260927 10:58:38.622174 20851 raft_consensus.cc:348] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.622442 20851 raft_consensus.cc:374] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:38.622501 20851 raft_consensus.cc:729] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae581ac897604d99a07c26fa2b7fba7e, State: Initialized, Role: FOLLOWER
I20260927 10:58:38.622927 20851 consensus_queue.cc:260] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.623845 20851 ts_tablet_manager.cc:1428] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:58:38.624135 20855 tablet_bootstrap.cc:654] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:38.624198 20699 heartbeater.cc:503] Master 127.1.46.254:45835 was elected leader, sending a full tablet report...
I20260927 10:58:38.625286 20855 log.cc:824] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:38.627219 20855 tablet_bootstrap.cc:492] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20: No bootstrap required, opened a new log
I20260927 10:58:38.627390 20855 ts_tablet_manager.cc:1397] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260927 10:58:38.629989 20855 raft_consensus.cc:348] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.630235 20855 raft_consensus.cc:374] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:38.630316 20855 raft_consensus.cc:729] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e167c6a4e5464e88cc8f8fe1239f20, State: Initialized, Role: FOLLOWER
I20260927 10:58:38.630743 20855 consensus_queue.cc:260] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.631368 20565 heartbeater.cc:503] Master 127.1.46.254:45835 was elected leader, sending a full tablet report...
I20260927 10:58:38.631574 20855 ts_tablet_manager.cc:1428] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
W20260927 10:58:38.631858 20566 tablet.cc:2369] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:38.694080 20854 raft_consensus.cc:482] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:38.694319 20854 raft_consensus.cc:504] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.695623 20854 leader_election.cc:288] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae581ac897604d99a07c26fa2b7fba7e (127.1.46.194:42709), 08e167c6a4e5464e88cc8f8fe1239f20 (127.1.46.193:44337)
I20260927 10:58:38.700527 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.695994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:35277 (local address 127.1.46.194:42709)
0927 10:58:38.696149 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.696155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.696172 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:58:38.696235 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.696239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.696297 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.696406 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:38.696703 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.697533 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.699201 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.699841 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.699996 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.700056 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.700313 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.700386 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:58:38.700527 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.696053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:59263 (local address 127.1.46.193:44337)
0927 10:58:38.696190 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.696194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.696389 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:58:38.696495 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.696501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.696559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.696652 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:38.696887 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.697615 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.699199 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.699855 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.699996 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.700056 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.700313 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.700386 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:58:38.700560 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.695847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:42709 (local address 127.1.46.195:35277)
0927 10:58:38.696016 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.696033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.696121 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.696427 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.696433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.696451 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:38.696571 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.696576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.697656 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.697659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.699057 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.699067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.699986 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.699990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.700158 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.700203 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.700279 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.700406 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":147}
I20260927 10:58:38.700568 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.695866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:44337 (local address 127.1.46.195:59263)
0927 10:58:38.696289 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.696300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:38.696406 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:38.696668 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:38.696674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:38.696693 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:38.696782 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.696786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.697735 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.697739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.699057 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:38.699067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.699987 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.699990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:38.700119 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.700201 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 10:58:38.700279 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:58:38.700406 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":122,"thread_start_us":120,"threads_started":1}
I20260927 10:58:38.701169 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab6cee2b8ee4cda9bc0798c77b8e35a" candidate_uuid: "21e12e5ca8044904a998f3926638ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" is_pre_election: true
I20260927 10:58:38.701517 20519 raft_consensus.cc:2454] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21e12e5ca8044904a998f3926638ddef in term 0.
I20260927 10:58:38.701491 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab6cee2b8ee4cda9bc0798c77b8e35a" candidate_uuid: "21e12e5ca8044904a998f3926638ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae581ac897604d99a07c26fa2b7fba7e" is_pre_election: true
I20260927 10:58:38.701751 20653 raft_consensus.cc:2454] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21e12e5ca8044904a998f3926638ddef in term 0.
I20260927 10:58:38.702000 20720 leader_election.cc:302] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e167c6a4e5464e88cc8f8fe1239f20, 21e12e5ca8044904a998f3926638ddef; no voters:
I20260927 10:58:38.702219 20854 raft_consensus.cc:2781] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:58:38.702311 20854 raft_consensus.cc:482] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:58:38.702360 20854 raft_consensus.cc:3037] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:38.703506 20854 raft_consensus.cc:504] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.703832 20854 leader_election.cc:288] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [CANDIDATE]: Term 1 election: Requested vote from peers ae581ac897604d99a07c26fa2b7fba7e (127.1.46.194:42709), 08e167c6a4e5464e88cc8f8fe1239f20 (127.1.46.193:44337)
I20260927 10:58:38.704103 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab6cee2b8ee4cda9bc0798c77b8e35a" candidate_uuid: "21e12e5ca8044904a998f3926638ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae581ac897604d99a07c26fa2b7fba7e"
I20260927 10:58:38.704246 20653 raft_consensus.cc:3037] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:38.704262 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab6cee2b8ee4cda9bc0798c77b8e35a" candidate_uuid: "21e12e5ca8044904a998f3926638ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e167c6a4e5464e88cc8f8fe1239f20"
I20260927 10:58:38.704361 20519 raft_consensus.cc:3037] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:38.705566 20519 raft_consensus.cc:2454] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e12e5ca8044904a998f3926638ddef in term 1.
I20260927 10:58:38.705566 20653 raft_consensus.cc:2454] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e12e5ca8044904a998f3926638ddef in term 1.
I20260927 10:58:38.705847 20720 leader_election.cc:302] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e167c6a4e5464e88cc8f8fe1239f20, 21e12e5ca8044904a998f3926638ddef; no voters:
I20260927 10:58:38.706070 20854 raft_consensus.cc:2781] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:38.706285 20854 raft_consensus.cc:686] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 1 LEADER]: Becoming Leader. State: Replica: 21e12e5ca8044904a998f3926638ddef, State: Running, Role: LEADER
I20260927 10:58:38.706698 20854 consensus_queue.cc:237] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } }
I20260927 10:58:38.709733 20374 catalog_manager.cc:5495] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef reported cstate change: term changed from 0 to 1, leader changed from <none> to 21e12e5ca8044904a998f3926638ddef (127.1.46.195). New cstate: current_term: 1 leader_uuid: "21e12e5ca8044904a998f3926638ddef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21e12e5ca8044904a998f3926638ddef" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 38145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20260927 10:58:38.748863 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.745539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.1.46.195:38145)
0927 10:58:38.745662 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:38.745665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:38.745687 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:58:38.745815 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:38.745818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:38.745868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:38.745951 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:58:38.746169 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.746649 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.747524 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:38.747740 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:38.747810 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:38.748062 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:58:38.748454 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:58:38.748537 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:38.748714 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:58:38.748760 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:58:38.754405 20653 raft_consensus.cc:1261] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 1 FOLLOWER]: Refusing update from remote peer 21e12e5ca8044904a998f3926638ddef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:38.754405 20519 raft_consensus.cc:1261] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 08e167c6a4e5464e88cc8f8fe1239f20 [term 1 FOLLOWER]: Refusing update from remote peer 21e12e5ca8044904a998f3926638ddef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:38.755100 20860 consensus_queue.cc:1035] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae581ac897604d99a07c26fa2b7fba7e" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:38.755250 20854 consensus_queue.cc:1035] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [LEADER]: Connected to new peer: Peer: permanent_uuid: "08e167c6a4e5464e88cc8f8fe1239f20" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:38.761700 20863 mvcc.cc:205] Tried to move back new op lower bound from 7333915520010850304 to 7333915519824957440. Current Snapshot: MvccSnapshot[applied={T|T < 7333915520010850304}]
I20260927 10:58:38.770128 20374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:38.770260 20374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:38.772476 20374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3902b2b38364410c88965ce838a75974: Sending DeleteTablet for 3 replicas of tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a
I20260927 10:58:38.773187 20633 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:38 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:33552
I20260927 10:58:38.773293 20499 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:38 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:54166
I20260927 10:58:38.773708 20873 tablet_replica.cc:330] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: stopping tablet replica
I20260927 10:58:38.773864 20767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:38 UTC) from {username='kudu', principal='kudu/127.1.46.254@KRBTEST.COM'} at 127.0.0.1:52194
I20260927 10:58:38.774201 20873 raft_consensus.cc:2229] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:38.774286 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20436
I20260927 10:58:38.774339 20875 tablet_replica.cc:330] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: stopping tablet replica
I20260927 10:58:38.774662 20873 raft_consensus.cc:2258] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:38.775691 20875 raft_consensus.cc:2229] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:58:38.776026 20875 raft_consensus.cc:2258] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:38.776360 20873 ts_tablet_manager.cc:1905] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:38.777385 20875 ts_tablet_manager.cc:1905] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:38.779212 20873 ts_tablet_manager.cc:1918] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:38.779377 20873 log.cc:1196] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/9ab6cee2b8ee4cda9bc0798c77b8e35a
I20260927 10:58:38.779783 20873 ts_tablet_manager.cc:1939] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P ae581ac897604d99a07c26fa2b7fba7e: Deleting consensus metadata
I20260927 10:58:38.780694 20875 ts_tablet_manager.cc:1918] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:38.780788 20875 log.cc:1196] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/9ab6cee2b8ee4cda9bc0798c77b8e35a
I20260927 10:58:38.781100 20875 ts_tablet_manager.cc:1939] T 9ab6cee2b8ee4cda9bc0798c77b8e35a P 21e12e5ca8044904a998f3926638ddef: Deleting consensus metadata
I20260927 10:58:38.781070 20361 catalog_manager.cc:4841] TS ae581ac897604d99a07c26fa2b7fba7e (127.1.46.194:42709): tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a (table test-table [id=6efb3852a60c46769c83ae86d0e751d4]) successfully deleted
I20260927 10:58:38.782209 20358 catalog_manager.cc:4841] TS 21e12e5ca8044904a998f3926638ddef (127.1.46.195:38145): tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a (table test-table [id=6efb3852a60c46769c83ae86d0e751d4]) successfully deleted
W20260927 10:58:38.783254 20358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:44337 (error 108)
I20260927 10:58:38.783524 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20570
I20260927 10:58:38.785199 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:38.784870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:44337 (local address 127.0.0.1:54180)
0927 10:58:38.785019 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:38.785097 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:44337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:58:38.785346 20358 catalog_manager.cc:4584] TS 08e167c6a4e5464e88cc8f8fe1239f20 (127.1.46.193:44337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ab6cee2b8ee4cda9bc0798c77b8e35a: Network error: Client connection negotiation failed: client connection to 127.1.46.193:44337: connect: Connection refused (error 111)
I20260927 10:58:38.789978 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20704
I20260927 10:58:38.795552 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20342
2026-09-27T10:58:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2264 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T10:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:38Z Disabled control of system clock
I20260927 10:58:38.826050 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40143
--webserver_interface=127.1.46.254
--webserver_port=0
--builtin_ntp_servers=127.1.46.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:58:38.933691 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:38.933914 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:38.933946 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:38.937685 20880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:38.937729 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:38.937743 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:38.937757 20880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:38.937770 20880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:38.942323 20880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:38.943248 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:38.944442 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:38.952572 20886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:38.952574 20885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:38.952787 20880 server_base.cc:1034] running on GCE node
W20260927 10:58:38.952605 20888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:38.953397 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:38.954382 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:38.955556 20880 hybrid_clock.cc:648] HybridClock initialized: now 1790506718955519 us; error 53 us; skew 500 ppm
I20260927 10:58:38.957281 20880 webserver.cc:460] Webserver started at http://127.1.46.254:46401/ using document root <none> and password file <none>
I20260927 10:58:38.957836 20880 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:38.957923 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:38.958139 20880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:38.960075 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "02880803a8d3489e872e6a35b74d27ac"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "b223e038cd6f432e1c75df54e71f97a7"
server_key_iv: "81fd74444e5d6f272c2299c8d1dae9fe"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.960582 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "02880803a8d3489e872e6a35b74d27ac"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "b223e038cd6f432e1c75df54e71f97a7"
server_key_iv: "81fd74444e5d6f272c2299c8d1dae9fe"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.964102 20880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:38.966235 20894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:38.967391 20880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:38.967519 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "02880803a8d3489e872e6a35b74d27ac"
format_stamp: "Formatted at 2026-09-27 10:58:38 on dist-test-slave-cffh"
server_key: "b223e038cd6f432e1c75df54e71f97a7"
server_key_iv: "81fd74444e5d6f272c2299c8d1dae9fe"
server_key_version: "encryptionkey@0"
I20260927 10:58:38.967614 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:38.997087 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:38.997743 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:38.997929 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:39.004920 20880 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:40143
I20260927 10:58:39.004925 20946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:40143 every 8 connection(s)
I20260927 10:58:39.005913 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:39.009068 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:39.011543 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20880
I20260927 10:58:39.011686 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:39.012029 1211 external_mini_cluster.cc:1428] Setting key 9809ca12e7456904365ff57ecd35bd8d
I20260927 10:58:39.014662 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac: Bootstrap starting.
I20260927 10:58:39.016916 20947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:39.017737 20947 log.cc:824] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:39.019819 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac: No bootstrap required, opened a new log
I20260927 10:58:39.021055 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.013428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.1.46.254:40143)
0927 10:58:39.013999 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.014013 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.014047 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:58:39.014691 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.014707 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.015111 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.015373 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:39.015734 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.016644 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.017888 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.018428 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.018542 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.019397 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:39.019401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:39.019413 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:39.019818 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:39.019900 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:39.019932 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:39.020052 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.020167 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.020404 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
I20260927 10:58:39.023550 20947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02880803a8d3489e872e6a35b74d27ac" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40143 } }
I20260927 10:58:39.023757 20947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:39.023839 20947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02880803a8d3489e872e6a35b74d27ac, State: Initialized, Role: FOLLOWER
I20260927 10:58:39.024279 20947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02880803a8d3489e872e6a35b74d27ac" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40143 } }
I20260927 10:58:39.024417 20947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:39.024495 20947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:39.024593 20947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:39.025398 20947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02880803a8d3489e872e6a35b74d27ac" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40143 } }
I20260927 10:58:39.025710 20947 leader_election.cc:302] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02880803a8d3489e872e6a35b74d27ac; no voters:
I20260927 10:58:39.025974 20947 leader_election.cc:288] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:39.026103 20952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:39.026337 20952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [term 1 LEADER]: Becoming Leader. State: Replica: 02880803a8d3489e872e6a35b74d27ac, State: Running, Role: LEADER
I20260927 10:58:39.026662 20952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02880803a8d3489e872e6a35b74d27ac" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40143 } }
I20260927 10:58:39.027074 20947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:39.028134 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02880803a8d3489e872e6a35b74d27ac. Latest consensus state: current_term: 1 leader_uuid: "02880803a8d3489e872e6a35b74d27ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02880803a8d3489e872e6a35b74d27ac" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40143 } } }
I20260927 10:58:39.028256 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:39.028609 20961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:39.029232 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02880803a8d3489e872e6a35b74d27ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02880803a8d3489e872e6a35b74d27ac" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40143 } } }
I20260927 10:58:39.029352 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:39.031150 20961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:39.035887 20961 catalog_manager.cc:1347] Generated new cluster ID: fd14c9b260c744caad5a3df7ed38d00b
I20260927 10:58:39.035939 20961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:39.059116 20961 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:39.060298 20961 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:39.076702 20961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac: Generated new TSK 0
I20260927 10:58:39.077407 20961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:39.089148 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.193:0
--local_ip_for_outbound_sockets=127.1.46.193
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40143
--builtin_ntp_servers=127.1.46.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:58:39.197353 20971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:39.197605 20971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:39.197642 20971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:39.203186 20971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:39.203236 20971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:39.203290 20971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.193
I20260927 10:58:39.208148 20971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.46.193
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.193
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:39.209169 20971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:39.210523 20971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:39.220116 20976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:39.220129 20977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:39.220329 20971 server_base.cc:1034] running on GCE node
W20260927 10:58:39.220118 20979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:39.220829 20971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:39.221436 20971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:39.222645 20971 hybrid_clock.cc:648] HybridClock initialized: now 1790506719222613 us; error 45 us; skew 500 ppm
I20260927 10:58:39.224365 20971 webserver.cc:460] Webserver started at http://127.1.46.193:43771/ using document root <none> and password file <none>
I20260927 10:58:39.224934 20971 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:39.225020 20971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:39.225232 20971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:39.227118 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data/instance:
uuid: "5626f9562ff346c09235fb3d6be6a62b"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "00610331711920e06fd80e64efa75ac3"
server_key_iv: "1225278480d2a72c055b27782d8b84a8"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.227792 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance:
uuid: "5626f9562ff346c09235fb3d6be6a62b"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "00610331711920e06fd80e64efa75ac3"
server_key_iv: "1225278480d2a72c055b27782d8b84a8"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.231303 20971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:39.233487 20985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.234692 20971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:39.234803 20971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/wal
uuid: "5626f9562ff346c09235fb3d6be6a62b"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "00610331711920e06fd80e64efa75ac3"
server_key_iv: "1225278480d2a72c055b27782d8b84a8"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.234907 20971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:39.258857 20971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:39.259470 20971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:39.259654 20971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:39.260219 20971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:39.261101 20971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:39.261168 20971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.261228 20971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:39.261270 20971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.270817 20971 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.193:33115
I20260927 10:58:39.270848 21098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.193:33115 every 8 connection(s)
I20260927 10:58:39.271782 20971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/data/info.pb
I20260927 10:58:39.275218 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 20971
I20260927 10:58:39.275405 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-0/wal/instance
I20260927 10:58:39.275768 1211 external_mini_cluster.cc:1428] Setting key 2a4b291b5b330aca45f2244ec58d70e9
I20260927 10:58:39.277990 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.194:0
--local_ip_for_outbound_sockets=127.1.46.194
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40143
--builtin_ntp_servers=127.1.46.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:58:39.282171 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.273699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.193:57375 (local address 127.1.46.254:40143)
0927 10:58:39.273838 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.273841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.274947 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 10:58:39.276504 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.276507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.276560 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.276675 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:39.277754 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.278355 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.279928 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.280333 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.280473 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.281681 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:39.281684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:39.281686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:39.281697 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:39.281716 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:39.281723 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:39.281780 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.281985 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.282054 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:58:39.283416 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.274089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40143 (local address 127.1.46.193:57375)
0927 10:58:39.274794 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.274832 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.276253 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.276871 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.276880 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.276977 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:39.277595 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.277615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.278481 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.278487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.279769 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.279779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.280534 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.280541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.280757 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.281392 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:39.281396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:39.281448 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:39.281524 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:39.281813 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:39.281824 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:39.281833 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.281965 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.282288 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":128,"threads_started":1}
I20260927 10:58:39.284783 21099 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40143
I20260927 10:58:39.285050 21099 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:39.285610 21099 heartbeater.cc:511] Master 127.1.46.254:40143 requested a full tablet report, sending...
I20260927 10:58:39.287281 20911 ts_manager.cc:194] Registered new tserver with Master: 5626f9562ff346c09235fb3d6be6a62b (127.1.46.193:33115)
I20260927 10:58:39.288537 20911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.193:57375
W20260927 10:58:39.384279 21102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:39.384519 21102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:39.384557 21102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:39.388540 21102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:39.388584 21102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:39.388638 21102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.194
I20260927 10:58:39.393661 21102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.46.194
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.194
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:39.394742 21102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:39.396067 21102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:39.405462 21108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:39.405519 21102 server_base.cc:1034] running on GCE node
W20260927 10:58:39.405475 21107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:39.405504 21110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:39.406069 21102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:39.406725 21102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:39.407879 21102 hybrid_clock.cc:648] HybridClock initialized: now 1790506719407851 us; error 40 us; skew 500 ppm
I20260927 10:58:39.409543 21102 webserver.cc:460] Webserver started at http://127.1.46.194:39135/ using document root <none> and password file <none>
I20260927 10:58:39.410072 21102 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:39.410130 21102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:39.410380 21102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:39.412320 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data/instance:
uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "be7797d3c219169ee8c86086da941834"
server_key_iv: "f02892ed98dffab022c48c2c7fd958d7"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.412847 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance:
uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "be7797d3c219169ee8c86086da941834"
server_key_iv: "f02892ed98dffab022c48c2c7fd958d7"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.416169 21102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:39.418463 21116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.419620 21102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:58:39.419781 21102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal
uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "be7797d3c219169ee8c86086da941834"
server_key_iv: "f02892ed98dffab022c48c2c7fd958d7"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.419875 21102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:39.430680 21102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:39.431209 21102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:39.431372 21102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:39.431890 21102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:39.432718 21102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:39.432762 21102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.432825 21102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:39.432873 21102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.441704 21102 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.194:46251
I20260927 10:58:39.441747 21229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.194:46251 every 8 connection(s)
I20260927 10:58:39.442473 21102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/data/info.pb
I20260927 10:58:39.444520 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21102
I20260927 10:58:39.444717 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal/instance
I20260927 10:58:39.445050 1211 external_mini_cluster.cc:1428] Setting key 945dbdf9e8333cb4c2e24aacf0be321e
I20260927 10:58:39.447328 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.46.195:0
--local_ip_for_outbound_sockets=127.1.46.195
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40143
--builtin_ntp_servers=127.1.46.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:58:39.452220 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.444232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:51291 (local address 127.1.46.254:40143)
0927 10:58:39.444401 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.444408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.445359 (+ 951us) server_negotiation.cc:407] Connection header received
0927 10:58:39.446928 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.446931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.446976 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.447054 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:39.448178 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.448899 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.450017 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.450438 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.450563 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.451630 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:39.451637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:39.451642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:39.451659 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:39.451687 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:39.451701 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:39.451786 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.451938 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.452011 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:58:39.452912 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.444539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40143 (local address 127.1.46.194:51291)
0927 10:58:39.445201 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.445248 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.446712 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.447235 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.447246 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.447344 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:39.448000 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.448019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.449036 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.449039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.449888 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.449896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.450621 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.450626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.450785 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.451379 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:39.451381 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:39.451417 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:39.451473 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:39.451799 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:39.451805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:39.451810 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.451905 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.452086 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":136,"threads_started":1}
I20260927 10:58:39.453989 21230 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40143
I20260927 10:58:39.454197 21230 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:39.454701 21230 heartbeater.cc:511] Master 127.1.46.254:40143 requested a full tablet report, sending...
I20260927 10:58:39.455726 20911 ts_manager.cc:194] Registered new tserver with Master: 78f0a7feddfa499ca5b2eb4a7b19d59d (127.1.46.194:46251)
I20260927 10:58:39.456581 20911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.194:51291
W20260927 10:58:39.556434 21233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:39.556679 21233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:39.556715 21233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:39.560645 21233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:39.560690 21233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:39.560739 21233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.46.195
I20260927 10:58:39.565362 21233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.46.195
--webserver_port=0
--tserver_master_addrs=127.1.46.254:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.46.195
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:39.566257 21233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:39.567601 21233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:39.576901 21241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:39.576925 21239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:39.576908 21238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:39.577203 21233 server_base.cc:1034] running on GCE node
I20260927 10:58:39.577826 21233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:39.578537 21233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:39.579833 21233 hybrid_clock.cc:648] HybridClock initialized: now 1790506719579815 us; error 29 us; skew 500 ppm
I20260927 10:58:39.581602 21233 webserver.cc:460] Webserver started at http://127.1.46.195:35011/ using document root <none> and password file <none>
I20260927 10:58:39.582187 21233 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:39.582276 21233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:39.582501 21233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:39.584559 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data/instance:
uuid: "214b319de51e4465a8d7ff13cc73c1e1"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "31def6225629a8a5dc70245dcd35a596"
server_key_iv: "683c36d2187d31575ab0b7d6e8a80230"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.585112 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance:
uuid: "214b319de51e4465a8d7ff13cc73c1e1"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "31def6225629a8a5dc70245dcd35a596"
server_key_iv: "683c36d2187d31575ab0b7d6e8a80230"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.588680 21233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:39.591054 21247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.592224 21233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:39.592361 21233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal
uuid: "214b319de51e4465a8d7ff13cc73c1e1"
format_stamp: "Formatted at 2026-09-27 10:58:39 on dist-test-slave-cffh"
server_key: "31def6225629a8a5dc70245dcd35a596"
server_key_iv: "683c36d2187d31575ab0b7d6e8a80230"
server_key_version: "encryptionkey@0"
I20260927 10:58:39.592469 21233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:39.610991 21233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:39.611601 21233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:39.611790 21233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:39.612319 21233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:58:39.613204 21233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:58:39.613276 21233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.613346 21233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:58:39.613390 21233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:39.622391 21233 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.195:40239
I20260927 10:58:39.622416 21360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.195:40239 every 8 connection(s)
I20260927 10:58:39.623289 21233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/data/info.pb
I20260927 10:58:39.623543 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21233
I20260927 10:58:39.623656 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal/instance
I20260927 10:58:39.624025 1211 external_mini_cluster.cc:1428] Setting key 1bf4dc087c03828ff65a0e77e71f8fbc
I20260927 10:58:39.632311 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.625209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.195:41883 (local address 127.1.46.254:40143)
0927 10:58:39.625338 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.625341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.626618 (+ 1277us) server_negotiation.cc:407] Connection header received
0927 10:58:39.628087 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.628090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.628132 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.628202 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:39.629138 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.629653 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.630707 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.630917 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.630993 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.631950 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:39.631953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:39.631955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:39.631965 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:39.631984 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:39.631991 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:39.632046 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.632167 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.632212 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:58:39.633098 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.625677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40143 (local address 127.1.46.195:41883)
0927 10:58:39.626426 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.626475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.627910 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.628390 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.628399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.628535 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:58:39.629025 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.629037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.629760 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.629763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.630613 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.630621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.631037 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.631040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.631178 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.631738 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:58:39.631739 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:58:39.631774 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:58:39.631828 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:58:39.632059 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:58:39.632065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:58:39.632069 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.632151 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.632333 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260927 10:58:39.634097 21361 heartbeater.cc:347] Connected to a master server at 127.1.46.254:40143
I20260927 10:58:39.634330 21361 heartbeater.cc:464] Registering TS with master...
I20260927 10:58:39.634848 21361 heartbeater.cc:511] Master 127.1.46.254:40143 requested a full tablet report, sending...
I20260927 10:58:39.635767 20911 ts_manager.cc:194] Registered new tserver with Master: 214b319de51e4465a8d7ff13cc73c1e1 (127.1.46.195:40239)
I20260927 10:58:39.636250 20911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.46.195:41883
I20260927 10:58:39.638458 1211 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:58:39.643916 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.640389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.1.46.254:40143)
0927 10:58:39.640492 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.640495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.640775 (+ 280us) server_negotiation.cc:407] Connection header received
0927 10:58:39.640838 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.640840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.640874 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.640936 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:39.641178 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.641645 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.642444 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.642826 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.642907 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.643590 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:39.643593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:39.643595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:39.643606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:39.643623 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:39.643630 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:39.643694 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.643777 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.643820 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:58:39.646853 20911 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:58:39.649729 20911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:58:39.663658 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.658670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.194:46251 (local address 127.0.0.1:36064)
0927 10:58:39.659491 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.659503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.659578 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.660094 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.660097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.660111 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:39.660196 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.660203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.661465 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.661471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.662225 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.662234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.663206 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.663208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.663281 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.663310 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.663383 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.663497 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":85,"threads_started":1}
I20260927 10:58:39.663760 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.658822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.1.46.194:46251)
0927 10:58:39.659162 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.659173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.659672 (+ 499us) server_negotiation.cc:407] Connection header received
0927 10:58:39.659764 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.659772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.659950 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.660115 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:39.660284 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.661364 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.662345 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.663091 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.663220 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.663416 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.663471 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.663562 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":81,"threads_started":1}
I20260927 10:58:39.663964 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.658204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:40239 (local address 127.0.0.1:55452)
0927 10:58:39.659204 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.659220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.659310 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.659816 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.659820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.659839 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:39.659936 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.659942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.661322 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.661326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.662519 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.662530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.663513 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.663516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.663596 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.663669 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.663766 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.663850 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":78,"thread_start_us":94,"threads_started":1}
I20260927 10:58:39.664341 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.658704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.1.46.195:40239)
0927 10:58:39.659357 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.659369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.659394 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:58:39.659468 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.659473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.659661 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.659816 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:39.660057 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.661207 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.662651 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.663395 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.663500 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.663674 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.664048 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.664151 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":96,"threads_started":1}
I20260927 10:58:39.665370 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.657865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33115 (local address 127.0.0.1:39366)
0927 10:58:39.658268 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.658327 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.658444 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.659246 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.659252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.659272 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:39.659408 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.659417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.660668 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.660674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.662526 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.662532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.664931 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.664935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.665034 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.665071 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.665162 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.665231 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":71,"thread_start_us":122,"threads_started":1}
I20260927 10:58:39.665789 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.658075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.1.46.193:33115)
0927 10:58:39.658502 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.658516 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.658542 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:58:39.658668 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.658674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.658898 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.659063 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:39.659544 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.660560 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.663768 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.664627 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.664745 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.664905 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.665571 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.665660 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":124,"threads_started":1}
I20260927 10:58:39.667377 21164 tablet_service.cc:1467] Processing CreateTablet for tablet 6909a71f2f7e461b8f5c69f87156de3a (DEFAULT_TABLE table=test-table [id=7db337b8ccc74b0aa74c7b0751dfc106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:39.667820 21295 tablet_service.cc:1467] Processing CreateTablet for tablet 6909a71f2f7e461b8f5c69f87156de3a (DEFAULT_TABLE table=test-table [id=7db337b8ccc74b0aa74c7b0751dfc106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:39.668006 21033 tablet_service.cc:1467] Processing CreateTablet for tablet 6909a71f2f7e461b8f5c69f87156de3a (DEFAULT_TABLE table=test-table [id=7db337b8ccc74b0aa74c7b0751dfc106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:58:39.668519 21164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6909a71f2f7e461b8f5c69f87156de3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:39.668794 21295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6909a71f2f7e461b8f5c69f87156de3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:39.668943 21033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6909a71f2f7e461b8f5c69f87156de3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:39.675040 21378 tablet_bootstrap.cc:492] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: Bootstrap starting.
I20260927 10:58:39.676468 21380 tablet_bootstrap.cc:492] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Bootstrap starting.
I20260927 10:58:39.677292 21378 tablet_bootstrap.cc:654] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:39.677999 21378 log.cc:824] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:39.678164 21380 tablet_bootstrap.cc:654] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:39.678923 21380 log.cc:824] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:39.678964 21379 tablet_bootstrap.cc:492] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Bootstrap starting.
I20260927 10:58:39.680632 21378 tablet_bootstrap.cc:492] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: No bootstrap required, opened a new log
I20260927 10:58:39.680658 21380 tablet_bootstrap.cc:492] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: No bootstrap required, opened a new log
I20260927 10:58:39.680784 21379 tablet_bootstrap.cc:654] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:39.680831 21380 ts_tablet_manager.cc:1397] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 10:58:39.680831 21378 ts_tablet_manager.cc:1397] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 10:58:39.681397 21379 log.cc:824] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:39.683019 21379 tablet_bootstrap.cc:492] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: No bootstrap required, opened a new log
I20260927 10:58:39.683219 21379 ts_tablet_manager.cc:1397] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 10:58:39.685118 21378 raft_consensus.cc:348] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.685118 21380 raft_consensus.cc:348] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.685384 21378 raft_consensus.cc:374] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:39.685384 21380 raft_consensus.cc:374] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:39.685448 21378 raft_consensus.cc:729] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5626f9562ff346c09235fb3d6be6a62b, State: Initialized, Role: FOLLOWER
I20260927 10:58:39.685448 21380 raft_consensus.cc:729] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b319de51e4465a8d7ff13cc73c1e1, State: Initialized, Role: FOLLOWER
I20260927 10:58:39.685951 21380 consensus_queue.cc:260] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.685951 21378 consensus_queue.cc:260] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.686547 21379 raft_consensus.cc:348] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.686827 21379 raft_consensus.cc:374] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:39.686896 21379 raft_consensus.cc:729] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f0a7feddfa499ca5b2eb4a7b19d59d, State: Initialized, Role: FOLLOWER
I20260927 10:58:39.686900 21361 heartbeater.cc:503] Master 127.1.46.254:40143 was elected leader, sending a full tablet report...
I20260927 10:58:39.686928 21380 ts_tablet_manager.cc:1428] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:58:39.687281 21099 heartbeater.cc:503] Master 127.1.46.254:40143 was elected leader, sending a full tablet report...
I20260927 10:58:39.687328 21379 consensus_queue.cc:260] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.688118 21230 heartbeater.cc:503] Master 127.1.46.254:40143 was elected leader, sending a full tablet report...
I20260927 10:58:39.688951 21378 ts_tablet_manager.cc:1428] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 10:58:39.689582 21379 ts_tablet_manager.cc:1428] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
W20260927 10:58:39.693243 21231 tablet.cc:2369] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:58:39.772977 21100 tablet.cc:2369] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:39.845934 21386 raft_consensus.cc:482] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:58:39.846138 21386 raft_consensus.cc:504] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.847249 21386 leader_election.cc:288] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5626f9562ff346c09235fb3d6be6a62b (127.1.46.193:33115), 214b319de51e4465a8d7ff13cc73c1e1 (127.1.46.195:40239)
I20260927 10:58:39.852252 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.847593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:55801 (local address 127.1.46.193:33115)
0927 10:58:39.847733 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.847739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.847822 (+ 83us) server_negotiation.cc:407] Connection header received
0927 10:58:39.847964 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.847966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.848033 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.848122 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:39.848476 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.849269 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.850995 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.851638 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.851762 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.851809 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.852061 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.852123 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:58:39.852252 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.847582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.46.194:60015 (local address 127.1.46.195:40239)
0927 10:58:39.847696 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.847701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.848052 (+ 351us) server_negotiation.cc:407] Connection header received
0927 10:58:39.848117 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.848120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.848162 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.848235 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:58:39.848517 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.849269 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.850995 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.851638 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.851769 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.851809 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.852061 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.852128 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:58:39.852283 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.847480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33115 (local address 127.1.46.194:55801)
0927 10:58:39.847705 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.847720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.847849 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.848157 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.848159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.848170 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:39.848308 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.848314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.849387 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.849391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.850858 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.850870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.851764 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.851769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.851910 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.851954 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.852028 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.852112 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":102}
I20260927 10:58:39.852289 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.847480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.195:40239 (local address 127.1.46.194:60015)
0927 10:58:39.847953 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.847970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:58:39.848043 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:58:39.848252 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:58:39.848254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:58:39.848268 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:58:39.848383 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.848409 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.849387 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.849391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.850858 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:58:39.850870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.851778 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.851780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:58:39.851917 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.851958 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:58:39.852028 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:58:39.852112 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":285,"thread_start_us":155,"threads_started":1}
I20260927 10:58:39.852946 21053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6909a71f2f7e461b8f5c69f87156de3a" candidate_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5626f9562ff346c09235fb3d6be6a62b" is_pre_election: true
I20260927 10:58:39.852952 21315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6909a71f2f7e461b8f5c69f87156de3a" candidate_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b319de51e4465a8d7ff13cc73c1e1" is_pre_election: true
I20260927 10:58:39.853287 21053 raft_consensus.cc:2454] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f0a7feddfa499ca5b2eb4a7b19d59d in term 0.
I20260927 10:58:39.853310 21315 raft_consensus.cc:2454] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f0a7feddfa499ca5b2eb4a7b19d59d in term 0.
I20260927 10:58:39.853804 21117 leader_election.cc:302] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 214b319de51e4465a8d7ff13cc73c1e1, 5626f9562ff346c09235fb3d6be6a62b, 78f0a7feddfa499ca5b2eb4a7b19d59d; no voters:
I20260927 10:58:39.854103 21386 raft_consensus.cc:2781] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:58:39.854187 21386 raft_consensus.cc:482] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:58:39.854241 21386 raft_consensus.cc:3037] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:39.855247 21386 raft_consensus.cc:504] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.855575 21386 leader_election.cc:288] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [CANDIDATE]: Term 1 election: Requested vote from peers 5626f9562ff346c09235fb3d6be6a62b (127.1.46.193:33115), 214b319de51e4465a8d7ff13cc73c1e1 (127.1.46.195:40239)
I20260927 10:58:39.855883 21053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6909a71f2f7e461b8f5c69f87156de3a" candidate_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5626f9562ff346c09235fb3d6be6a62b"
I20260927 10:58:39.855911 21315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6909a71f2f7e461b8f5c69f87156de3a" candidate_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b319de51e4465a8d7ff13cc73c1e1"
I20260927 10:58:39.855991 21053 raft_consensus.cc:3037] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:39.856070 21315 raft_consensus.cc:3037] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:39.856989 21053 raft_consensus.cc:2454] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f0a7feddfa499ca5b2eb4a7b19d59d in term 1.
I20260927 10:58:39.857129 21315 raft_consensus.cc:2454] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f0a7feddfa499ca5b2eb4a7b19d59d in term 1.
I20260927 10:58:39.857283 21117 leader_election.cc:302] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5626f9562ff346c09235fb3d6be6a62b, 78f0a7feddfa499ca5b2eb4a7b19d59d; no voters:
I20260927 10:58:39.858096 21386 raft_consensus.cc:2781] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:39.858347 21386 raft_consensus.cc:686] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 1 LEADER]: Becoming Leader. State: Replica: 78f0a7feddfa499ca5b2eb4a7b19d59d, State: Running, Role: LEADER
I20260927 10:58:39.858733 21386 consensus_queue.cc:237] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } }
I20260927 10:58:39.862879 20910 catalog_manager.cc:5495] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d reported cstate change: term changed from 0 to 1, leader changed from <none> to 78f0a7feddfa499ca5b2eb4a7b19d59d (127.1.46.194). New cstate: current_term: 1 leader_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f0a7feddfa499ca5b2eb4a7b19d59d" member_type: VOTER last_known_addr { host: "127.1.46.194" port: 46251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 } health_report { overall_health: UNKNOWN } } }
W20260927 10:58:39.874396 21362 tablet.cc:2369] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:58:39.897707 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.893704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.1.46.194:46251)
0927 10:58:39.893862 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:39.893867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:39.893883 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:58:39.893972 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:39.893976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:39.894040 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:39.894147 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:58:39.894389 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.895166 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.896136 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:39.896435 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:39.896563 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:39.896886 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:58:39.897300 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:58:39.897435 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:39.897471 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 10:58:39.897521 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:58:39.904233 21315 raft_consensus.cc:1261] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 1 FOLLOWER]: Refusing update from remote peer 78f0a7feddfa499ca5b2eb4a7b19d59d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:39.904400 21053 raft_consensus.cc:1261] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b [term 1 FOLLOWER]: Refusing update from remote peer 78f0a7feddfa499ca5b2eb4a7b19d59d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:58:39.905025 21386 consensus_queue.cc:1035] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "214b319de51e4465a8d7ff13cc73c1e1" member_type: VOTER last_known_addr { host: "127.1.46.195" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:39.905226 21388 consensus_queue.cc:1035] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5626f9562ff346c09235fb3d6be6a62b" member_type: VOTER last_known_addr { host: "127.1.46.193" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:58:39.913021 21391 mvcc.cc:205] Tried to move back new op lower bound from 7333915524720025600 to 7333915524544679936. Current Snapshot: MvccSnapshot[applied={T|T < 7333915524720025600}]
I20260927 10:58:39.915346 21392 mvcc.cc:205] Tried to move back new op lower bound from 7333915524720025600 to 7333915524544679936. Current Snapshot: MvccSnapshot[applied={T|T < 7333915524720025600 or (T in {7333915524720025600})}]
I20260927 10:58:39.922523 20910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:39.922747 20910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:58:39.925096 20910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02880803a8d3489e872e6a35b74d27ac: Sending DeleteTablet for 3 replicas of tablet 6909a71f2f7e461b8f5c69f87156de3a
I20260927 10:58:39.925859 21033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6909a71f2f7e461b8f5c69f87156de3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:39 UTC) from {username='slave'} at 127.0.0.1:39366
I20260927 10:58:39.925858 21295 tablet_service.cc:1514] Processing DeleteTablet for tablet 6909a71f2f7e461b8f5c69f87156de3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:39 UTC) from {username='slave'} at 127.0.0.1:55452
I20260927 10:58:39.926330 21400 tablet_replica.cc:330] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: stopping tablet replica
I20260927 10:58:39.926431 21164 tablet_service.cc:1514] Processing DeleteTablet for tablet 6909a71f2f7e461b8f5c69f87156de3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:58:39 UTC) from {username='slave'} at 127.0.0.1:36064
I20260927 10:58:39.926661 21401 tablet_replica.cc:330] T 6909a71f2f7e461b8f5c69f87156de3a P 5626f9562ff346c09235fb3d6be6a62b: stopping tablet replica
I20260927 10:58:39.926853 21402 tablet_replica.cc:330] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: stopping tablet replica
I20260927 10:58:39.926865 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20971
I20260927 10:58:39.928417 21402 raft_consensus.cc:2229] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:58:39.928407 21400 raft_consensus.cc:2229] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:58:39.928778 21402 raft_consensus.cc:2258] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:39.929083 21400 raft_consensus.cc:2258] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:58:39.930190 21402 ts_tablet_manager.cc:1905] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:39.930608 21400 ts_tablet_manager.cc:1905] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:58:39.932914 21402 ts_tablet_manager.cc:1918] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:39.933010 21402 log.cc:1196] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-1/wal/wals/6909a71f2f7e461b8f5c69f87156de3a
I20260927 10:58:39.933332 21402 ts_tablet_manager.cc:1939] T 6909a71f2f7e461b8f5c69f87156de3a P 78f0a7feddfa499ca5b2eb4a7b19d59d: Deleting consensus metadata
I20260927 10:58:39.933499 21400 ts_tablet_manager.cc:1918] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:58:39.933634 21400 log.cc:1196] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Deleting WAL directory at /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790506472751632-1211-0/minicluster-data/ts-2/wal/wals/6909a71f2f7e461b8f5c69f87156de3a
I20260927 10:58:39.934006 21400 ts_tablet_manager.cc:1939] T 6909a71f2f7e461b8f5c69f87156de3a P 214b319de51e4465a8d7ff13cc73c1e1: Deleting consensus metadata
I20260927 10:58:39.934470 20895 catalog_manager.cc:4841] TS 78f0a7feddfa499ca5b2eb4a7b19d59d (127.1.46.194:46251): tablet 6909a71f2f7e461b8f5c69f87156de3a (table test-table [id=7db337b8ccc74b0aa74c7b0751dfc106]) successfully deleted
W20260927 10:58:39.934814 20895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.46.193:33115 (error 108)
I20260927 10:58:39.935010 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21102
I20260927 10:58:39.936124 20896 catalog_manager.cc:4841] TS 214b319de51e4465a8d7ff13cc73c1e1 (127.1.46.195:40239): tablet 6909a71f2f7e461b8f5c69f87156de3a (table test-table [id=7db337b8ccc74b0aa74c7b0751dfc106]) successfully deleted
I20260927 10:58:39.936313 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:39.935873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.193:33115 (local address 127.0.0.1:39372)
0927 10:58:39.936035 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 10:58:39.936123 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.193:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:58:39.936472 20895 catalog_manager.cc:4584] TS 5626f9562ff346c09235fb3d6be6a62b (127.1.46.193:33115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6909a71f2f7e461b8f5c69f87156de3a: Network error: Client connection negotiation failed: client connection to 127.1.46.193:33115: connect: Connection refused (error 111)
I20260927 10:58:39.940821 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21233
I20260927 10:58:39.946287 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 20880
2026-09-27T10:58:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1149 ms)
[----------] 6 tests from EncryptionPolicyTest (10930 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:39 dist-test-slave-cffh krb5kdc[21406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:39 dist-test-slave-cffh krb5kdc[21406](info): set up 2 sockets
Sep 27 10:58:39 dist-test-slave-cffh krb5kdc[21406](info): commencing operation
krb5kdc: starting...
W20260927 10:58:40.970830 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:41 dist-test-slave-cffh krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:41.083854 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:41.189319 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:41.189568 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:41.189607 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:41.193663 21419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:41.193734 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:41.193791 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:41.193822 21419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:41.193850 21419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:41.199016 21419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:41.199958 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:41.200918 21419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:41.209506 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:41.209568 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:41.209604 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:41.210063 21419 server_base.cc:1034] running on GCE node
I20260927 10:58:41.210624 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:41.211757 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:41.212947 21419 hybrid_clock.cc:648] HybridClock initialized: now 1790506721212931 us; error 29 us; skew 500 ppm
Sep 27 10:58:41 dist-test-slave-cffh krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:41.215580 21419 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:41.216320 21419 webserver.cc:460] Webserver started at http://127.0.0.1:40513/ using document root <none> and password file <none>
I20260927 10:58:41.216763 21419 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:41.216813 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:41.216953 21419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:41.218808 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "40515f9f72ef4cfcbe79a792c97a1bcf"
format_stamp: "Formatted at 2026-09-27 10:58:41 on dist-test-slave-cffh"
server_key: "a359c1ed64cc934ec369c8e18112ab4b"
server_key_iv: "8300afd34b0bdcfa773df90909de3376"
server_key_version: "encryptionkey@0"
I20260927 10:58:41.219296 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "40515f9f72ef4cfcbe79a792c97a1bcf"
format_stamp: "Formatted at 2026-09-27 10:58:41 on dist-test-slave-cffh"
server_key: "a359c1ed64cc934ec369c8e18112ab4b"
server_key_iv: "8300afd34b0bdcfa773df90909de3376"
server_key_version: "encryptionkey@0"
I20260927 10:58:41.222553 21419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:41.224720 21434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:41.225780 21419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:58:41.225916 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "40515f9f72ef4cfcbe79a792c97a1bcf"
format_stamp: "Formatted at 2026-09-27 10:58:41 on dist-test-slave-cffh"
server_key: "a359c1ed64cc934ec369c8e18112ab4b"
server_key_iv: "8300afd34b0bdcfa773df90909de3376"
server_key_version: "encryptionkey@0"
I20260927 10:58:41.226007 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:41.242550 21419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:41.245110 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:41.245299 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:41.251868 21419 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42667
I20260927 10:58:41.251932 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42667 every 8 connection(s)
I20260927 10:58:41.252816 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:41.255458 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:41.259117 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21419
I20260927 10:58:41.259261 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:41.259531 1211 external_mini_cluster.cc:1428] Setting key 8973ebc74ee6b964e943e2cbab388161
I20260927 10:58:41.261010 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf: Bootstrap starting.
I20260927 10:58:41.263584 21487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:41.264454 21487 log.cc:824] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:41.266171 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf: No bootstrap required, opened a new log
Sep 27 10:58:41 dist-test-slave-cffh krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:58:41.270249 21487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260927 10:58:41.270515 21487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:41.270597 21487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40515f9f72ef4cfcbe79a792c97a1bcf, State: Initialized, Role: FOLLOWER
I20260927 10:58:41.271091 21487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260927 10:58:41.271271 21487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:41.271341 21487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:41.271436 21487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:41.272465 21487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260927 10:58:41.272841 21487 leader_election.cc:302] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40515f9f72ef4cfcbe79a792c97a1bcf; no voters:
I20260927 10:58:41.273175 21487 leader_election.cc:288] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:41.273274 21492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:41.273489 21492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [term 1 LEADER]: Becoming Leader. State: Replica: 40515f9f72ef4cfcbe79a792c97a1bcf, State: Running, Role: LEADER
I20260927 10:58:41.273768 21492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260927 10:58:41.274412 21487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:41.275260 21493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } } }
I20260927 10:58:41.275388 21493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:41.275722 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:41.260924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41611 (local address 127.0.0.1:42667)
0927 10:58:41.261382 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:41.261400 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:41.261439 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:58:41.262272 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:41.262296 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:41.262722 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:41.263050 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:41.263747 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:41.265087 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:41.265989 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:41.266650 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:41.266786 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:41.269906 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:41.269919 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:41.269934 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:41.269961 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:41.272104 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:41.272610 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:41.272616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:41.272621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:41.272747 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:41.273121 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:41.273127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:41.273131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:41.273612 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:41.274360 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:41.274522 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:58:41.274750 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20260927 10:58:41.275717 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40515f9f72ef4cfcbe79a792c97a1bcf. Latest consensus state: current_term: 1 leader_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40515f9f72ef4cfcbe79a792c97a1bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } } }
I20260927 10:58:41.275856 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:41.275738 21501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:41.278995 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:41.279422 21501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:41.285178 21501 catalog_manager.cc:1347] Generated new cluster ID: 2fecce22d13c40c08e24aec6296e1a62
I20260927 10:58:41.285235 21501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:41.291081 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:41.281101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42137 (local address 127.0.0.1:42667)
0927 10:58:41.281205 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:41.281209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:41.281508 (+ 299us) server_negotiation.cc:407] Connection header received
0927 10:58:41.281647 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:41.281652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:41.281701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:41.281785 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:41.282382 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:41.282995 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:41.284024 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:41.284510 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:41.284652 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:41.287601 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:41.287605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:41.287608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:41.287635 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:41.289315 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:41.289808 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:41.289811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:41.289813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:41.289897 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:41.290219 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:41.290222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:41.290224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:41.290418 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:41.290553 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:41.290840 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 10:58:41.290933 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:58:41.296880 21501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:41.297698 21501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:41.303519 21501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40515f9f72ef4cfcbe79a792c97a1bcf: Generated new TSK 0
I20260927 10:58:41.304133 21501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:41.308525 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21419
W20260927 10:58:41.313941 21404 connection.cc:535] client connection to 127.0.0.1:42667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42667): Connection reset by peer (error 104)
2026-09-27T10:58:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:41 dist-test-slave-cffh krb5kdc[21518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:41 dist-test-slave-cffh krb5kdc[21518](info): set up 2 sockets
Sep 27 10:58:41 dist-test-slave-cffh krb5kdc[21518](info): commencing operation
krb5kdc: starting...
W20260927 10:58:42.326769 1211 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:42 dist-test-slave-cffh krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:42.445565 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:42.552416 21531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:42.552706 21531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:42.552793 21531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:42.556833 21531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:42.556910 21531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:42.556947 21531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:42.557001 21531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:42.557040 21531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:42.562013 21531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:42.563017 21531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:42.563990 21531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:42.571949 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:42.571949 21536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:42.572108 21531 server_base.cc:1034] running on GCE node
W20260927 10:58:42.572026 21537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:42.572865 21531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:42.573961 21531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:42.575111 21531 hybrid_clock.cc:648] HybridClock initialized: now 1790506722575097 us; error 26 us; skew 500 ppm
Sep 27 10:58:42 dist-test-slave-cffh krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:42.577746 21531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:42.578506 21531 webserver.cc:460] Webserver started at http://127.0.0.1:34917/ using document root <none> and password file <none>
I20260927 10:58:42.579058 21531 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:42.579124 21531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:42.579284 21531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:42.581195 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "5bfd4894017d4eb18cbb7229e4d35e59"
format_stamp: "Formatted at 2026-09-27 10:58:42 on dist-test-slave-cffh"
server_key: "818a0b8c614c8e055d7607eaddd3d907"
server_key_iv: "f759ed8ce72f4d41a03a9678cdd0cc9c"
server_key_version: "encryptionkey@0"
I20260927 10:58:42.581687 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "5bfd4894017d4eb18cbb7229e4d35e59"
format_stamp: "Formatted at 2026-09-27 10:58:42 on dist-test-slave-cffh"
server_key: "818a0b8c614c8e055d7607eaddd3d907"
server_key_iv: "f759ed8ce72f4d41a03a9678cdd0cc9c"
server_key_version: "encryptionkey@0"
I20260927 10:58:42.585152 21531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:58:42.587379 21546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:42.588507 21531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:42.588649 21531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "5bfd4894017d4eb18cbb7229e4d35e59"
format_stamp: "Formatted at 2026-09-27 10:58:42 on dist-test-slave-cffh"
server_key: "818a0b8c614c8e055d7607eaddd3d907"
server_key_iv: "f759ed8ce72f4d41a03a9678cdd0cc9c"
server_key_version: "encryptionkey@0"
I20260927 10:58:42.588757 21531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:42.607756 21531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:42.610462 21531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:42.610709 21531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:42.617873 21531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41803
I20260927 10:58:42.617882 21598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41803 every 8 connection(s)
I20260927 10:58:42.618824 21531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:42.621208 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21531
I20260927 10:58:42.621354 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:42.621647 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:42.621670 1211 external_mini_cluster.cc:1428] Setting key aba021a64b66a42f775c2dc0f7f9f32d
I20260927 10:58:42.628188 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59: Bootstrap starting.
I20260927 10:58:42.630493 21599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:42.630823 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:42.622981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34589 (local address 127.0.0.1:41803)
0927 10:58:42.623415 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:42.623433 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:42.623467 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:58:42.624128 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:42.624145 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:42.624459 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:42.624767 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:42.625171 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:42.626168 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:42.627389 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:42.628008 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:42.628136 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:42.629083 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:42.629088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:42.629101 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:42.629463 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:42.629514 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:42.629538 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:42.629679 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:42.629870 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 10:58:42.630008 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20260927 10:58:42.631237 21599 log.cc:824] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:42.633059 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59: No bootstrap required, opened a new log
I20260927 10:58:42.633708 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:42.637398 21599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41803 } }
I20260927 10:58:42.637640 21599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:42.637702 21599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bfd4894017d4eb18cbb7229e4d35e59, State: Initialized, Role: FOLLOWER
I20260927 10:58:42.638152 21599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41803 } }
I20260927 10:58:42.638279 21599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:42.638352 21599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:42.638443 21599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:42.639444 21599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41803 } }
I20260927 10:58:42.639782 21599 leader_election.cc:302] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bfd4894017d4eb18cbb7229e4d35e59; no voters:
I20260927 10:58:42.640169 21599 leader_election.cc:288] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:42.640292 21609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:42.640501 21609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [term 1 LEADER]: Becoming Leader. State: Replica: 5bfd4894017d4eb18cbb7229e4d35e59, State: Running, Role: LEADER
I20260927 10:58:42.640637 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:42.636081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53867 (local address 127.0.0.1:41803)
0927 10:58:42.636206 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:42.636210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:42.636421 (+ 211us) server_negotiation.cc:407] Connection header received
0927 10:58:42.636515 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:42.636518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:42.636557 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:42.636644 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:42.636897 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:42.637575 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:42.638599 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:42.638910 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:42.639012 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:42.640090 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:42.640101 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:42.640104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:42.640115 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:42.640138 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:42.640149 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:42.640250 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:42.640403 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:58:42.640435 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:58:42.640811 21609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41803 } }
I20260927 10:58:42.641371 21599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:42.641891 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bfd4894017d4eb18cbb7229e4d35e59. Latest consensus state: current_term: 1 leader_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41803 } } }
I20260927 10:58:42.641989 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:42.642369 21610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfd4894017d4eb18cbb7229e4d35e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41803 } } }
I20260927 10:58:42.642480 21610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:42.642732 21614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:42.645848 21614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:42.651262 21614 catalog_manager.cc:1347] Generated new cluster ID: f49f93386390423cbeaff525996fa7a2
I20260927 10:58:42.651322 21614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:42.665501 21614 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:42.666356 21614 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:42.674998 21614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bfd4894017d4eb18cbb7229e4d35e59: Generated new TSK 0
I20260927 10:58:42.675704 21614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:42.684759 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21531
W20260927 10:58:42.690234 21516 connection.cc:535] client connection to 127.0.0.1:41803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41803): Connection reset by peer (error 104)
2026-09-27T10:58:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:42 dist-test-slave-cffh krb5kdc[21630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:42 dist-test-slave-cffh krb5kdc[21630](info): set up 2 sockets
Sep 27 10:58:42 dist-test-slave-cffh krb5kdc[21630](info): commencing operation
krb5kdc: starting...
W20260927 10:58:43.710726 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:43 dist-test-slave-cffh krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:43.823473 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:43.927240 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:43.927500 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:43.927570 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:43.931526 21643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:43.931602 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:43.931639 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:43.931694 21643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:43.931738 21643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:43.936689 21643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:43.937672 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:43.938699 21643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:43.946696 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:43.946707 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:43.946833 21643 server_base.cc:1034] running on GCE node
W20260927 10:58:43.946718 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:43.947551 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:43.948557 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:43.949774 21643 hybrid_clock.cc:648] HybridClock initialized: now 1790506723949763 us; error 23 us; skew 500 ppm
Sep 27 10:58:43 dist-test-slave-cffh krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:43.952544 21643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:43.953473 21643 webserver.cc:460] Webserver started at http://127.0.0.1:46747/ using document root <none> and password file <none>
I20260927 10:58:43.953995 21643 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:43.954082 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:43.954299 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:43.956321 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "a694ed661ae54569861778a1c23ab4ec"
format_stamp: "Formatted at 2026-09-27 10:58:43 on dist-test-slave-cffh"
server_key: "f275734b9d8830148a118d80c4c31c82"
server_key_iv: "4ef588dd6a0033c07c6e61ce919c788e"
server_key_version: "encryptionkey@0"
I20260927 10:58:43.956889 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "a694ed661ae54569861778a1c23ab4ec"
format_stamp: "Formatted at 2026-09-27 10:58:43 on dist-test-slave-cffh"
server_key: "f275734b9d8830148a118d80c4c31c82"
server_key_iv: "4ef588dd6a0033c07c6e61ce919c788e"
server_key_version: "encryptionkey@0"
I20260927 10:58:43.960438 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:43.962802 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:43.963940 21643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:58:43.964073 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "a694ed661ae54569861778a1c23ab4ec"
format_stamp: "Formatted at 2026-09-27 10:58:43 on dist-test-slave-cffh"
server_key: "f275734b9d8830148a118d80c4c31c82"
server_key_iv: "4ef588dd6a0033c07c6e61ce919c788e"
server_key_version: "encryptionkey@0"
I20260927 10:58:43.964179 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:43.968665 21643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:43.971789 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:43.971972 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:43.979115 21643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38413
I20260927 10:58:43.979137 21710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38413 every 8 connection(s)
I20260927 10:58:43.979967 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:43.982491 21711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:43.987846 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec: Bootstrap starting.
I20260927 10:58:43.988623 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21643
I20260927 10:58:43.988786 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:43.989101 1211 external_mini_cluster.cc:1428] Setting key d85f5961b7a21a3ea03ba7aaeee936a8
I20260927 10:58:43.990769 21711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:43.991451 21711 log.cc:824] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:43.993391 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec: No bootstrap required, opened a new log
I20260927 10:58:43.993914 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:43.990239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47107 (local address 127.0.0.1:38413)
0927 10:58:43.990684 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:43.990697 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:43.990736 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:58:43.991358 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:43.991373 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:43.991675 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:43.991987 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:43.992756 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:43.992762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:43.992773 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:43.992783 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:43.992830 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:43.992849 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:43.992959 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:43.993062 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 10:58:43.993218 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260927 10:58:43.996743 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:43.997434 21711 raft_consensus.cc:348] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694ed661ae54569861778a1c23ab4ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260927 10:58:43.997622 21711 raft_consensus.cc:374] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:43.997691 21711 raft_consensus.cc:729] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a694ed661ae54569861778a1c23ab4ec, State: Initialized, Role: FOLLOWER
I20260927 10:58:43.998109 21711 consensus_queue.cc:260] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694ed661ae54569861778a1c23ab4ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260927 10:58:43.998234 21711 raft_consensus.cc:388] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:43.998317 21711 raft_consensus.cc:482] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:43.998416 21711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:43.999565 21711 raft_consensus.cc:504] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694ed661ae54569861778a1c23ab4ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260927 10:58:43.999900 21711 leader_election.cc:302] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a694ed661ae54569861778a1c23ab4ec; no voters:
I20260927 10:58:44.000214 21711 leader_election.cc:288] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:44.000320 21721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:44.000429 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:43.998726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38791 (local address 127.0.0.1:38413)
0927 10:58:43.998859 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:43.998863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:43.999094 (+ 231us) server_negotiation.cc:407] Connection header received
0927 10:58:43.999201 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:43.999208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:43.999233 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:43.999294 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:43.999994 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:43.999998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:44.000002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:44.000014 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:44.000043 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:44.000055 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:44.000127 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:44.000229 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 10:58:44.000262 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:58:44.000718 21721 raft_consensus.cc:686] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [term 1 LEADER]: Becoming Leader. State: Replica: a694ed661ae54569861778a1c23ab4ec, State: Running, Role: LEADER
I20260927 10:58:44.001016 21721 consensus_queue.cc:237] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694ed661ae54569861778a1c23ab4ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260927 10:58:44.001395 21711 sys_catalog.cc:564] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:44.002156 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader a694ed661ae54569861778a1c23ab4ec. Latest consensus state: current_term: 1 leader_uuid: "a694ed661ae54569861778a1c23ab4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694ed661ae54569861778a1c23ab4ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } } }
I20260927 10:58:44.002283 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:44.002912 21722 sys_catalog.cc:455] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a694ed661ae54569861778a1c23ab4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694ed661ae54569861778a1c23ab4ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } } }
I20260927 10:58:44.003022 21722 sys_catalog.cc:458] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:44.003307 21730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:44.005961 21730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:44.011050 21730 catalog_manager.cc:1347] Generated new cluster ID: 2bbf30d2824c48cf84130422a82ff23d
I20260927 10:58:44.011104 21730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:44.020591 21730 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:44.020860 21730 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:44.032356 21730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a694ed661ae54569861778a1c23ab4ec: Generated new TSK 0
I20260927 10:58:44.033059 21730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:44.036114 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21643
2026-09-27T10:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:44 dist-test-slave-cffh krb5kdc[21742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:44 dist-test-slave-cffh krb5kdc[21742](info): set up 2 sockets
Sep 27 10:58:44 dist-test-slave-cffh krb5kdc[21742](info): commencing operation
krb5kdc: starting...
W20260927 10:58:45.058915 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:45 dist-test-slave-cffh krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:45.174911 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:45.280911 21755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:45.281172 21755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:45.281252 21755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:45.285063 21755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:45.285153 21755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:45.285192 21755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:45.285249 21755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:45.285290 21755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:45.290258 21755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:45.291242 21755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:45.292228 21755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:45.300280 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:45.300346 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:45.300310 21763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:45.300459 21755 server_base.cc:1034] running on GCE node
I20260927 10:58:45.301115 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:45.302155 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:45.303313 21755 hybrid_clock.cc:648] HybridClock initialized: now 1790506725303300 us; error 23 us; skew 500 ppm
Sep 27 10:58:45 dist-test-slave-cffh krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:45.309190 21755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:45.310102 21755 webserver.cc:460] Webserver started at http://127.0.0.1:41849/ using document root <none> and password file <none>
I20260927 10:58:45.310704 21755 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:45.310788 21755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:45.311010 21755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:45.312916 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "27ce6943f7d947d880b0b7e09872cd96"
format_stamp: "Formatted at 2026-09-27 10:58:45 on dist-test-slave-cffh"
server_key: "f595aecdb5b947217aec5854c0059640"
server_key_iv: "2c06eb38252e23326bbe3e12286763c3"
server_key_version: "encryptionkey@0"
I20260927 10:58:45.313445 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "27ce6943f7d947d880b0b7e09872cd96"
format_stamp: "Formatted at 2026-09-27 10:58:45 on dist-test-slave-cffh"
server_key: "f595aecdb5b947217aec5854c0059640"
server_key_iv: "2c06eb38252e23326bbe3e12286763c3"
server_key_version: "encryptionkey@0"
I20260927 10:58:45.316954 21755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:45.319164 21770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:45.320302 21755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:45.320429 21755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "27ce6943f7d947d880b0b7e09872cd96"
format_stamp: "Formatted at 2026-09-27 10:58:45 on dist-test-slave-cffh"
server_key: "f595aecdb5b947217aec5854c0059640"
server_key_iv: "2c06eb38252e23326bbe3e12286763c3"
server_key_version: "encryptionkey@0"
I20260927 10:58:45.320533 21755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:45.330118 21755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:45.332715 21755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:45.332908 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:45.339542 21755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44661
I20260927 10:58:45.339565 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44661 every 8 connection(s)
I20260927 10:58:45.340436 21755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:45.342903 21823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:45.347927 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96: Bootstrap starting.
I20260927 10:58:45.349990 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21755
I20260927 10:58:45.350138 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:45.350116 21823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:45.350410 1211 external_mini_cluster.cc:1428] Setting key dfbf84e79f936d0b50c6727eea2fbc6a
I20260927 10:58:45.350982 21823 log.cc:824] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:45.353044 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96: No bootstrap required, opened a new log
I20260927 10:58:45.357143 21823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ce6943f7d947d880b0b7e09872cd96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260927 10:58:45.357381 21823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:45.357504 21823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ce6943f7d947d880b0b7e09872cd96, State: Initialized, Role: FOLLOWER
I20260927 10:58:45.357966 21823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ce6943f7d947d880b0b7e09872cd96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260927 10:58:45.358111 21823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:45.358175 21823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:45.358266 21823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 10:58:45 dist-test-slave-cffh krb5kdc[21742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:58:45.359385 21823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ce6943f7d947d880b0b7e09872cd96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260927 10:58:45.359704 21823 leader_election.cc:302] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27ce6943f7d947d880b0b7e09872cd96; no voters:
I20260927 10:58:45.360023 21823 leader_election.cc:288] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:45.360132 21828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:45.360373 21828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [term 1 LEADER]: Becoming Leader. State: Replica: 27ce6943f7d947d880b0b7e09872cd96, State: Running, Role: LEADER
I20260927 10:58:45.360737 21828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ce6943f7d947d880b0b7e09872cd96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260927 10:58:45.361291 21823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:45.361917 21829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27ce6943f7d947d880b0b7e09872cd96. Latest consensus state: current_term: 1 leader_uuid: "27ce6943f7d947d880b0b7e09872cd96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ce6943f7d947d880b0b7e09872cd96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } } }
I20260927 10:58:45.362114 21829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:45.362323 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27ce6943f7d947d880b0b7e09872cd96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ce6943f7d947d880b0b7e09872cd96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } } }
I20260927 10:58:45.362421 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:45.362694 21837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:45.365710 21837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:45.365845 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:45.351498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49827 (local address 127.0.0.1:44661)
0927 10:58:45.351896 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:45.351913 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:45.351951 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:58:45.352753 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:45.352774 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:45.353163 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:45.353513 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:45.354158 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:45.355617 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:45.356416 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:45.357058 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:45.357220 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:45.360054 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:45.360061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:45.360078 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:45.360106 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:45.362192 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:45.362746 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:45.362752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:45.362757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:45.362906 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:45.363275 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:45.363279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:45.363282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:45.363804 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:45.364000 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:45.364742 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 10:58:45.365004 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260927 10:58:45.369163 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:45.371431 21837 catalog_manager.cc:1347] Generated new cluster ID: 7a3b7558a833411997c6caba1ea10652
I20260927 10:58:45.371503 21837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:45.379943 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:45.371248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43257 (local address 127.0.0.1:44661)
0927 10:58:45.371329 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:45.371334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:45.371811 (+ 477us) server_negotiation.cc:407] Connection header received
0927 10:58:45.371932 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:45.371935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:45.371976 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:45.372056 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:45.372763 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:45.373534 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:45.374544 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:45.374954 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:45.375064 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:45.376506 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:45.376510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:45.376513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:45.376538 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:45.378153 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:45.378563 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:45.378600 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:45.378601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:45.378679 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:45.378941 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:45.378944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:45.378946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:45.379157 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:45.379281 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:45.379582 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 10:58:45.379685 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 10:58:45.387526 21837 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:45.388458 21837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:45.392545 21837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27ce6943f7d947d880b0b7e09872cd96: Generated new TSK 0
I20260927 10:58:45.393162 21837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:45.397747 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21755
2026-09-27T10:58:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:45 dist-test-slave-cffh krb5kdc[21854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:45 dist-test-slave-cffh krb5kdc[21854](info): set up 2 sockets
Sep 27 10:58:45 dist-test-slave-cffh krb5kdc[21854](info): commencing operation
krb5kdc: starting...
W20260927 10:58:46.418839 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:46 dist-test-slave-cffh krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:46.529330 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:46.636217 21867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:46.636463 21867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:46.636504 21867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:46.640527 21867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:46.640581 21867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:46.640600 21867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:46.640616 21867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:46.640631 21867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:46.645744 21867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:46.646736 21867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:46.647706 21867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:46.655805 21875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:46.655850 21873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:46.655850 21872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:46.656222 21867 server_base.cc:1034] running on GCE node
I20260927 10:58:46.656708 21867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:46.657702 21867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:46.658856 21867 hybrid_clock.cc:648] HybridClock initialized: now 1790506726658845 us; error 25 us; skew 500 ppm
Sep 27 10:58:46 dist-test-slave-cffh krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:46.661618 21867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:46.662412 21867 webserver.cc:460] Webserver started at http://127.0.0.1:38431/ using document root <none> and password file <none>
I20260927 10:58:46.662977 21867 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:46.663061 21867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:46.663271 21867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:46.665128 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "1c1a3185f2f043268fea4969704ae6c6"
format_stamp: "Formatted at 2026-09-27 10:58:46 on dist-test-slave-cffh"
server_key: "cbcd547eccfa053f6e4213f375f33517"
server_key_iv: "406bbf40e79a19f6f4a8c7c44449b3f3"
server_key_version: "encryptionkey@0"
I20260927 10:58:46.665649 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "1c1a3185f2f043268fea4969704ae6c6"
format_stamp: "Formatted at 2026-09-27 10:58:46 on dist-test-slave-cffh"
server_key: "cbcd547eccfa053f6e4213f375f33517"
server_key_iv: "406bbf40e79a19f6f4a8c7c44449b3f3"
server_key_version: "encryptionkey@0"
I20260927 10:58:46.669135 21867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:58:46.671442 21882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:46.672577 21867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:46.672700 21867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "1c1a3185f2f043268fea4969704ae6c6"
format_stamp: "Formatted at 2026-09-27 10:58:46 on dist-test-slave-cffh"
server_key: "cbcd547eccfa053f6e4213f375f33517"
server_key_iv: "406bbf40e79a19f6f4a8c7c44449b3f3"
server_key_version: "encryptionkey@0"
I20260927 10:58:46.672811 21867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:46.689025 21867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:46.691620 21867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:46.691805 21867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:46.699082 21867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40339
I20260927 10:58:46.699105 21934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40339 every 8 connection(s)
I20260927 10:58:46.699968 21867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:46.702526 21935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:46.704524 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21867
I20260927 10:58:46.704662 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:46.704977 1211 external_mini_cluster.cc:1428] Setting key e1e77e54e6d02f15446839d95fd91f3d
I20260927 10:58:46.708802 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6: Bootstrap starting.
I20260927 10:58:46.711371 21935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:46.712189 21935 log.cc:824] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:46.713532 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:46.706218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39945 (local address 127.0.0.1:40339)
0927 10:58:46.706608 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:46.706630 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:46.706663 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:58:46.707271 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:46.707286 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:46.707595 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:46.707846 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:46.708226 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:46.709220 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:46.710402 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:46.710902 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:46.711015 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:46.711888 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:46.711893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:46.711905 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:46.712244 (+ 339us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:46.712291 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:46.712342 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:46.712430 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:46.712618 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 10:58:46.712817 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260927 10:58:46.714100 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6: No bootstrap required, opened a new log
I20260927 10:58:46.716243 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:46.718459 21935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a3185f2f043268fea4969704ae6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40339 } }
I20260927 10:58:46.718763 21935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:46.718832 21935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1a3185f2f043268fea4969704ae6c6, State: Initialized, Role: FOLLOWER
I20260927 10:58:46.719300 21935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a3185f2f043268fea4969704ae6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40339 } }
I20260927 10:58:46.719493 21935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:46.719568 21935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:46.719655 21935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:46.720678 21935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a3185f2f043268fea4969704ae6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40339 } }
I20260927 10:58:46.721009 21935 leader_election.cc:302] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c1a3185f2f043268fea4969704ae6c6; no voters:
I20260927 10:58:46.721364 21935 leader_election.cc:288] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:46.721447 21945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:46.721623 21945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c1a3185f2f043268fea4969704ae6c6, State: Running, Role: LEADER
I20260927 10:58:46.721961 21945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a3185f2f043268fea4969704ae6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40339 } }
I20260927 10:58:46.722595 21935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:46.723302 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:46.718409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47301 (local address 127.0.0.1:40339)
0927 10:58:46.718541 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:46.718545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:46.718802 (+ 257us) server_negotiation.cc:407] Connection header received
0927 10:58:46.718973 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:46.718975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:46.719013 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:46.719109 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:46.719365 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:46.720040 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:46.721092 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:46.721429 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:46.721554 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:46.722688 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:46.722693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:46.722696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:46.722708 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:46.722736 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:46.722753 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:46.722858 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:46.723122 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 10:58:46.723175 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:58:46.723677 21947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c1a3185f2f043268fea4969704ae6c6. Latest consensus state: current_term: 1 leader_uuid: "1c1a3185f2f043268fea4969704ae6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a3185f2f043268fea4969704ae6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40339 } } }
I20260927 10:58:46.723812 21947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:46.724165 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c1a3185f2f043268fea4969704ae6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a3185f2f043268fea4969704ae6c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40339 } } }
I20260927 10:58:46.724318 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6 [sys.catalog]: This master's current role is: LEADER
W20260927 10:58:46.727243 21960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:58:46.727306 21960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:58:46.727384 21954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:46.727990 21954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:46.733330 21954 catalog_manager.cc:1347] Generated new cluster ID: 1bf49534b41b4ab990e87e94a191c146
I20260927 10:58:46.733399 21954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:46.759896 21954 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:46.760811 21954 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:46.774744 21954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c1a3185f2f043268fea4969704ae6c6: Generated new TSK 0
I20260927 10:58:46.775382 21954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:46.781375 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21867
2026-09-27T10:58:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:46 dist-test-slave-cffh krb5kdc[21966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:46 dist-test-slave-cffh krb5kdc[21966](info): set up 2 sockets
Sep 27 10:58:46 dist-test-slave-cffh krb5kdc[21966](info): commencing operation
krb5kdc: starting...
W20260927 10:58:47.806850 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:47 dist-test-slave-cffh krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:47.923362 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:48.027558 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:48.027820 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:48.027910 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:48.031881 21979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:48.031955 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:48.032017 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:48.032069 21979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:48.032106 21979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:48.037185 21979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:48.038106 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:48.039139 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:48.047039 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:48.047063 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:48.047215 21979 server_base.cc:1034] running on GCE node
W20260927 10:58:48.047102 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:48.047848 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:48.048882 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:48.050040 21979 hybrid_clock.cc:648] HybridClock initialized: now 1790506728050012 us; error 40 us; skew 500 ppm
Sep 27 10:58:48 dist-test-slave-cffh krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:48.052695 21979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:48.053524 21979 webserver.cc:460] Webserver started at http://127.0.0.1:46733/ using document root <none> and password file <none>
I20260927 10:58:48.054000 21979 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:48.054052 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:48.054199 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:48.056108 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "5303de64ec30482da8bea16a1fec6c11"
format_stamp: "Formatted at 2026-09-27 10:58:48 on dist-test-slave-cffh"
server_key: "ffd2bcb0f3899870a6cbf328d39e10f5"
server_key_iv: "eac4ce1846ba5f1403d7eb8d07636ad9"
server_key_version: "encryptionkey@0"
I20260927 10:58:48.056569 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "5303de64ec30482da8bea16a1fec6c11"
format_stamp: "Formatted at 2026-09-27 10:58:48 on dist-test-slave-cffh"
server_key: "ffd2bcb0f3899870a6cbf328d39e10f5"
server_key_iv: "eac4ce1846ba5f1403d7eb8d07636ad9"
server_key_version: "encryptionkey@0"
I20260927 10:58:48.059907 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:58:48.062080 21994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:48.063313 21979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:58:48.063452 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "5303de64ec30482da8bea16a1fec6c11"
format_stamp: "Formatted at 2026-09-27 10:58:48 on dist-test-slave-cffh"
server_key: "ffd2bcb0f3899870a6cbf328d39e10f5"
server_key_iv: "eac4ce1846ba5f1403d7eb8d07636ad9"
server_key_version: "encryptionkey@0"
I20260927 10:58:48.063560 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:48.067752 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:48.070840 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:48.071019 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:48.077564 21979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34065
I20260927 10:58:48.077570 22046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34065 every 8 connection(s)
I20260927 10:58:48.078431 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:48.081135 22047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:48.086354 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11: Bootstrap starting.
I20260927 10:58:48.088471 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 21979
I20260927 10:58:48.088617 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:48.088786 22047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:48.088927 1211 external_mini_cluster.cc:1428] Setting key d5f8969ad9a3b25a8ce1d902f9b43adf
I20260927 10:58:48.089529 22047 log.cc:824] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:48.091734 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11: No bootstrap required, opened a new log
I20260927 10:58:48.094165 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:48.089961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57447 (local address 127.0.0.1:34065)
0927 10:58:48.090429 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:48.090448 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:48.090489 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:58:48.091308 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:48.091330 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:48.091672 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:48.092017 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:48.092761 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:48.092767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:48.092782 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:48.092794 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:48.092855 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:48.092879 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:48.092990 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:48.093130 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 10:58:48.093282 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":133,"threads_started":1}
I20260927 10:58:48.095958 22047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5303de64ec30482da8bea16a1fec6c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34065 } }
I20260927 10:58:48.096205 22047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:48.096290 22047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5303de64ec30482da8bea16a1fec6c11, State: Initialized, Role: FOLLOWER
I20260927 10:58:48.096753 22047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5303de64ec30482da8bea16a1fec6c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34065 } }
I20260927 10:58:48.096911 22047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:48.096976 22047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:48.097031 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:48.097080 22047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:48.097893 22047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5303de64ec30482da8bea16a1fec6c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34065 } }
I20260927 10:58:48.098186 22047 leader_election.cc:302] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5303de64ec30482da8bea16a1fec6c11; no voters:
I20260927 10:58:48.098474 22047 leader_election.cc:288] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:48.098553 22056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:48.099285 22056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [term 1 LEADER]: Becoming Leader. State: Replica: 5303de64ec30482da8bea16a1fec6c11, State: Running, Role: LEADER
I20260927 10:58:48.099625 22056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5303de64ec30482da8bea16a1fec6c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34065 } }
I20260927 10:58:48.099802 22047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:48.100940 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:48.099119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46521 (local address 127.0.0.1:34065)
0927 10:58:48.099232 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:48.099244 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:48.099407 (+ 163us) server_negotiation.cc:407] Connection header received
0927 10:58:48.099583 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:48.099586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:48.099620 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:48.099706 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:48.100439 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:48.100442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:48.100445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:48.100456 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:48.100482 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:48.100493 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:48.100579 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:48.100756 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:58:48.100800 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 10:58:48.101636 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5303de64ec30482da8bea16a1fec6c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5303de64ec30482da8bea16a1fec6c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34065 } } }
I20260927 10:58:48.101764 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:48.101965 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5303de64ec30482da8bea16a1fec6c11. Latest consensus state: current_term: 1 leader_uuid: "5303de64ec30482da8bea16a1fec6c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5303de64ec30482da8bea16a1fec6c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34065 } } }
I20260927 10:58:48.102036 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11 [sys.catalog]: This master's current role is: LEADER
W20260927 10:58:48.104805 22072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:58:48.104880 22072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:58:48.104936 22071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:48.105794 22071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:48.111186 22071 catalog_manager.cc:1347] Generated new cluster ID: 1ccada47316f4d949d2509399dc4d9ef
I20260927 10:58:48.111250 22071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:48.143193 22071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:48.143554 22071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:48.151355 22071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5303de64ec30482da8bea16a1fec6c11: Generated new TSK 0
I20260927 10:58:48.152148 22071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:48.167369 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 21979
2026-09-27T10:58:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:48 dist-test-slave-cffh krb5kdc[22078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:48 dist-test-slave-cffh krb5kdc[22078](info): set up 2 sockets
Sep 27 10:58:48 dist-test-slave-cffh krb5kdc[22078](info): commencing operation
krb5kdc: starting...
W20260927 10:58:49.186858 1211 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:49 dist-test-slave-cffh krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:49.302136 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:49.407269 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:49.407501 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:49.407539 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:49.411370 22091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:49.411417 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:49.411437 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:49.411451 22091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:49.411465 22091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:49.416231 22091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:49.417079 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:49.417948 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:49.425843 22099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:49.425855 22097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:49.425900 22091 server_base.cc:1034] running on GCE node
W20260927 10:58:49.425900 22096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:49.426824 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:49.427863 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:49.429034 22091 hybrid_clock.cc:648] HybridClock initialized: now 1790506729429011 us; error 47 us; skew 500 ppm
Sep 27 10:58:49 dist-test-slave-cffh krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:49.431706 22091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:49.432502 22091 webserver.cc:460] Webserver started at http://127.0.0.1:40525/ using document root <none> and password file <none>
I20260927 10:58:49.432981 22091 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:49.433034 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:49.433184 22091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:49.435101 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "a67fc0a98aee49ac8968eeb4d372f537"
format_stamp: "Formatted at 2026-09-27 10:58:49 on dist-test-slave-cffh"
server_key: "f1ccd497a3a48c989dd42e70514e3307"
server_key_iv: "3d7dd9176fd0406340ded7ef80b63fdb"
server_key_version: "encryptionkey@0"
I20260927 10:58:49.435575 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "a67fc0a98aee49ac8968eeb4d372f537"
format_stamp: "Formatted at 2026-09-27 10:58:49 on dist-test-slave-cffh"
server_key: "f1ccd497a3a48c989dd42e70514e3307"
server_key_iv: "3d7dd9176fd0406340ded7ef80b63fdb"
server_key_version: "encryptionkey@0"
I20260927 10:58:49.439047 22091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:58:49.441371 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:49.442520 22091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:49.442694 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "a67fc0a98aee49ac8968eeb4d372f537"
format_stamp: "Formatted at 2026-09-27 10:58:49 on dist-test-slave-cffh"
server_key: "f1ccd497a3a48c989dd42e70514e3307"
server_key_iv: "3d7dd9176fd0406340ded7ef80b63fdb"
server_key_version: "encryptionkey@0"
I20260927 10:58:49.442785 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:49.456039 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:49.458446 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:49.458650 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:49.465032 22091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35471
I20260927 10:58:49.465061 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35471 every 8 connection(s)
I20260927 10:58:49.465853 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:49.467475 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22091
I20260927 10:58:49.467622 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:49.468022 1211 external_mini_cluster.cc:1428] Setting key dbe6febd898ea6b2b7fe045a7b64192d
I20260927 10:58:49.469009 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:49.475332 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537: Bootstrap starting.
Sep 27 10:58:49 dist-test-slave-cffh krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:58:49.477777 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:49.478660 22159 log.cc:824] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:49.480726 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537: No bootstrap required, opened a new log
I20260927 10:58:49.481938 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:49.469335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:36313 (local address 127.0.0.1:35471)
0927 10:58:49.469694 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:49.469718 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:49.469750 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:58:49.470337 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:49.470351 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:49.470714 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:49.470964 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:49.471943 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:49.472897 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:49.474039 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:49.474499 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:49.474680 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:49.477293 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:49.477297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:49.477309 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:49.477331 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:49.479109 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:49.479720 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:49.479724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:49.479728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:49.479822 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:49.480234 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:49.480236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:49.480238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:49.480555 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:49.480708 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:49.481082 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 10:58:49.481289 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":130,"threads_started":1}
I20260927 10:58:49.484716 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67fc0a98aee49ac8968eeb4d372f537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260927 10:58:49.484982 22159 raft_consensus.cc:374] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:49.485008 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:49.485069 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67fc0a98aee49ac8968eeb4d372f537, State: Initialized, Role: FOLLOWER
I20260927 10:58:49.485541 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67fc0a98aee49ac8968eeb4d372f537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260927 10:58:49.485713 22159 raft_consensus.cc:388] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:49.485786 22159 raft_consensus.cc:482] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:49.485900 22159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:49.487064 22159 raft_consensus.cc:504] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67fc0a98aee49ac8968eeb4d372f537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260927 10:58:49.487442 22159 leader_election.cc:302] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a67fc0a98aee49ac8968eeb4d372f537; no voters:
I20260927 10:58:49.487784 22159 leader_election.cc:288] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:49.487931 22169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:49.488140 22169 raft_consensus.cc:686] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [term 1 LEADER]: Becoming Leader. State: Replica: a67fc0a98aee49ac8968eeb4d372f537, State: Running, Role: LEADER
I20260927 10:58:49.488525 22169 consensus_queue.cc:237] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67fc0a98aee49ac8968eeb4d372f537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260927 10:58:49.489044 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:49.489744 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a67fc0a98aee49ac8968eeb4d372f537. Latest consensus state: current_term: 1 leader_uuid: "a67fc0a98aee49ac8968eeb4d372f537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67fc0a98aee49ac8968eeb4d372f537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } } }
I20260927 10:58:49.489915 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:49.490142 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a67fc0a98aee49ac8968eeb4d372f537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67fc0a98aee49ac8968eeb4d372f537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } } }
I20260927 10:58:49.490240 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:49.490419 22178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:49.493207 22178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:49.497987 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:49.487348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:55433 (local address 127.0.0.1:35471)
0927 10:58:49.487454 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:49.487458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:49.487893 (+ 435us) server_negotiation.cc:407] Connection header received
0927 10:58:49.488116 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:49.488123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:49.488173 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:49.488252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:49.489206 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:49.490032 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:49.491353 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:49.491749 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:49.491857 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:49.493987 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:49.493992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:49.493995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:49.494021 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:49.495858 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:49.496464 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:49.496467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:49.496469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:49.496552 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:49.497037 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:49.497041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:49.497043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:49.497254 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:49.497405 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:49.497737 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 10:58:49.497834 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:58:49.498802 22178 catalog_manager.cc:1347] Generated new cluster ID: 5b132acfdbc54233883c446032380634
I20260927 10:58:49.498911 22178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:49.504809 22178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:49.505752 22178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:49.510586 22178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a67fc0a98aee49ac8968eeb4d372f537: Generated new TSK 0
I20260927 10:58:49.511345 22178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:49.522444 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22091
W20260927 10:58:49.527578 22076 connection.cc:535] client connection to 127.0.0.1:35471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35471): Connection reset by peer (error 104)
2026-09-27T10:58:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:49 dist-test-slave-cffh krb5kdc[22190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:49 dist-test-slave-cffh krb5kdc[22190](info): set up 2 sockets
Sep 27 10:58:49 dist-test-slave-cffh krb5kdc[22190](info): commencing operation
krb5kdc: starting...
W20260927 10:58:50.550897 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:50 dist-test-slave-cffh krb5kdc[22190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:50.667354 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:50.772504 22203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:50.772753 22203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:50.772794 22203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:50.776733 22203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:50.776783 22203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:50.776803 22203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:50.776819 22203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:50.776878 22203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:50.781996 22203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:50.782948 22203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:50.783907 22203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:50.792284 22209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:50.792303 22211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:50.792309 22208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:50.792439 22203 server_base.cc:1034] running on GCE node
I20260927 10:58:50.793135 22203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:50.794204 22203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:50.795411 22203 hybrid_clock.cc:648] HybridClock initialized: now 1790506730795394 us; error 33 us; skew 500 ppm
Sep 27 10:58:50 dist-test-slave-cffh krb5kdc[22190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:50.803766 22203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:50.804641 22203 webserver.cc:460] Webserver started at http://127.0.0.1:41591/ using document root <none> and password file <none>
I20260927 10:58:50.805200 22203 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:50.805294 22203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:50.805526 22203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:50.807588 22203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "73bd5c44af06464580d6ef161901fbd3"
format_stamp: "Formatted at 2026-09-27 10:58:50 on dist-test-slave-cffh"
server_key: "37ccc913afa0e1fb6c7d89e47bbe5355"
server_key_iv: "f72d22924e992d0b4133bfd22d22da17"
server_key_version: "encryptionkey@0"
I20260927 10:58:50.808166 22203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "73bd5c44af06464580d6ef161901fbd3"
format_stamp: "Formatted at 2026-09-27 10:58:50 on dist-test-slave-cffh"
server_key: "37ccc913afa0e1fb6c7d89e47bbe5355"
server_key_iv: "f72d22924e992d0b4133bfd22d22da17"
server_key_version: "encryptionkey@0"
I20260927 10:58:50.812424 22203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:50.814777 22218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:50.815914 22203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:58:50.816052 22203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "73bd5c44af06464580d6ef161901fbd3"
format_stamp: "Formatted at 2026-09-27 10:58:50 on dist-test-slave-cffh"
server_key: "37ccc913afa0e1fb6c7d89e47bbe5355"
server_key_iv: "f72d22924e992d0b4133bfd22d22da17"
server_key_version: "encryptionkey@0"
I20260927 10:58:50.816156 22203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:50.831017 22203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:50.833478 22203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:50.833653 22203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:50.840211 22203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45183
I20260927 10:58:50.840233 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45183 every 8 connection(s)
I20260927 10:58:50.841069 22203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:50.842731 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22203
I20260927 10:58:50.842865 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:50.843144 1211 external_mini_cluster.cc:1428] Setting key 1de6e339858acbd14657a3ce5194797f
I20260927 10:58:50.843878 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:50.850185 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3: Bootstrap starting.
I20260927 10:58:50.852686 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:50.844416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:38163 (local address 127.0.0.1:45183)
0927 10:58:50.844820 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:50.844837 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:50.844885 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:58:50.845709 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:50.845729 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:50.846150 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:50.846488 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:50.846894 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:50.848259 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:50.849174 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:50.849837 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:50.849993 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:50.850885 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:50.850892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:50.850905 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:50.851362 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:50.851428 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:50.851461 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:50.851588 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:50.851753 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 10:58:50.851876 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260927 10:58:50.852706 22271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:50.853444 22271 log.cc:824] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:50.855427 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3: No bootstrap required, opened a new log
I20260927 10:58:50.855556 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:50.858944 22271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bd5c44af06464580d6ef161901fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260927 10:58:50.859171 22271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:50.859251 22271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73bd5c44af06464580d6ef161901fbd3, State: Initialized, Role: FOLLOWER
I20260927 10:58:50.859699 22271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bd5c44af06464580d6ef161901fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260927 10:58:50.859838 22271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:50.859905 22271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:50.859997 22271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:50.860983 22271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bd5c44af06464580d6ef161901fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260927 10:58:50.861304 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:50.857402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:58601 (local address 127.0.0.1:45183)
0927 10:58:50.857517 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:50.857521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:50.857717 (+ 196us) server_negotiation.cc:407] Connection header received
0927 10:58:50.857849 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:50.857855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:50.857893 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:50.857973 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:50.858219 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:50.858772 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:50.859683 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:50.859975 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:50.860068 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:50.860828 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:50.860833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:50.860836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:50.860848 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:50.860883 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:50.860894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:50.860983 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:50.861129 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 10:58:50.861161 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:58:50.861323 22271 leader_election.cc:302] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73bd5c44af06464580d6ef161901fbd3; no voters:
I20260927 10:58:50.861722 22271 leader_election.cc:288] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:50.861824 22281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:50.862026 22281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [term 1 LEADER]: Becoming Leader. State: Replica: 73bd5c44af06464580d6ef161901fbd3, State: Running, Role: LEADER
I20260927 10:58:50.862353 22281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bd5c44af06464580d6ef161901fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260927 10:58:50.862891 22271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:50.864033 22283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73bd5c44af06464580d6ef161901fbd3. Latest consensus state: current_term: 1 leader_uuid: "73bd5c44af06464580d6ef161901fbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bd5c44af06464580d6ef161901fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } } }
I20260927 10:58:50.864033 22282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73bd5c44af06464580d6ef161901fbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bd5c44af06464580d6ef161901fbd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } } }
I20260927 10:58:50.864210 22283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:50.864259 22282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:50.864624 22291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:50.867341 22291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:50.872319 22291 catalog_manager.cc:1347] Generated new cluster ID: 0b79c80c0b2b430289b32533f879a02f
I20260927 10:58:50.872376 22291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:50.887305 22291 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:50.888167 22291 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:50.891788 22291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73bd5c44af06464580d6ef161901fbd3: Generated new TSK 0
I20260927 10:58:50.892392 22291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:50.900831 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22203
W20260927 10:58:50.905783 22188 connection.cc:535] client connection to 127.0.0.1:45183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45183): Connection reset by peer (error 104)
2026-09-27T10:58:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:50 dist-test-slave-cffh krb5kdc[22302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:50 dist-test-slave-cffh krb5kdc[22302](info): set up 2 sockets
Sep 27 10:58:50 dist-test-slave-cffh krb5kdc[22302](info): commencing operation
krb5kdc: starting...
W20260927 10:58:51.923007 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:51 dist-test-slave-cffh krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:52.038812 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:52.143118 22315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:52.143330 22315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:52.143368 22315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:52.147405 22315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:52.147462 22315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:52.147498 22315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:52.147514 22315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:52.147529 22315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:52.152539 22315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:52.153436 22315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:52.154309 22315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:52.162420 22321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:52.162420 22320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:52.162441 22323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:52.162799 22315 server_base.cc:1034] running on GCE node
I20260927 10:58:52.163350 22315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:52.164383 22315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:52.165563 22315 hybrid_clock.cc:648] HybridClock initialized: now 1790506732165531 us; error 45 us; skew 500 ppm
Sep 27 10:58:52 dist-test-slave-cffh krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:52.168258 22315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:52.169119 22315 webserver.cc:460] Webserver started at http://127.0.0.1:41193/ using document root <none> and password file <none>
I20260927 10:58:52.169643 22315 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:52.169732 22315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:52.169961 22315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:52.171955 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "1d95e433cc3c474ba0d0bdc37f170b4d"
format_stamp: "Formatted at 2026-09-27 10:58:52 on dist-test-slave-cffh"
server_key: "12d60d039a5c21254589a78bbc475b03"
server_key_iv: "b825883a09723d79facdef801c22542e"
server_key_version: "encryptionkey@0"
I20260927 10:58:52.172497 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "1d95e433cc3c474ba0d0bdc37f170b4d"
format_stamp: "Formatted at 2026-09-27 10:58:52 on dist-test-slave-cffh"
server_key: "12d60d039a5c21254589a78bbc475b03"
server_key_iv: "b825883a09723d79facdef801c22542e"
server_key_version: "encryptionkey@0"
I20260927 10:58:52.175932 22315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:58:52.178150 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:52.179378 22315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:52.179524 22315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "1d95e433cc3c474ba0d0bdc37f170b4d"
format_stamp: "Formatted at 2026-09-27 10:58:52 on dist-test-slave-cffh"
server_key: "12d60d039a5c21254589a78bbc475b03"
server_key_iv: "b825883a09723d79facdef801c22542e"
server_key_version: "encryptionkey@0"
I20260927 10:58:52.179626 22315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:52.183876 22315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:52.186635 22315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:52.186822 22315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:52.193387 22315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34845
I20260927 10:58:52.193447 22382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34845 every 8 connection(s)
I20260927 10:58:52.194275 22315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:52.196852 22383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:52.201756 22383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d: Bootstrap starting.
I20260927 10:58:52.203899 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22315
I20260927 10:58:52.203914 22383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:52.204035 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:52.204331 1211 external_mini_cluster.cc:1428] Setting key 38fc2729b0760b0f6fa38da1966d7129
I20260927 10:58:52.204687 22383 log.cc:824] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:52.206730 22383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d: No bootstrap required, opened a new log
I20260927 10:58:52.209124 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:52.205323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:53469 (local address 127.0.0.1:34845)
0927 10:58:52.205731 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:52.205748 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:52.205788 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:58:52.206538 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:52.206558 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:52.206896 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:52.207230 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:52.207872 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:52.207878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:52.207890 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:52.207902 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:52.207961 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:52.207984 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:52.208084 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:52.208169 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 10:58:52.208315 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260927 10:58:52.210170 22383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260927 10:58:52.210354 22383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:52.210430 22383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d95e433cc3c474ba0d0bdc37f170b4d, State: Initialized, Role: FOLLOWER
I20260927 10:58:52.210868 22383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260927 10:58:52.210983 22383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:52.211097 22383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:52.211217 22383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:52.212064 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:52.212143 22383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260927 10:58:52.212469 22383 leader_election.cc:302] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d95e433cc3c474ba0d0bdc37f170b4d; no voters:
I20260927 10:58:52.212744 22383 leader_election.cc:288] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:52.212847 22388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:52.213059 22388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [term 1 LEADER]: Becoming Leader. State: Replica: 1d95e433cc3c474ba0d0bdc37f170b4d, State: Running, Role: LEADER
I20260927 10:58:52.213440 22388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260927 10:58:52.213902 22383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:52.214852 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d95e433cc3c474ba0d0bdc37f170b4d. Latest consensus state: current_term: 1 leader_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260927 10:58:52.215045 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:52.215348 22389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d95e433cc3c474ba0d0bdc37f170b4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260927 10:58:52.215451 22389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:52.217794 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:52.214444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:45231 (local address 127.0.0.1:34845)
0927 10:58:52.214591 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:52.214595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:52.214970 (+ 375us) server_negotiation.cc:407] Connection header received
0927 10:58:52.216458 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:52.216464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:52.216498 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:52.216545 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:52.217391 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:52.217395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:52.217397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:52.217408 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:52.217433 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:52.217445 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:52.217544 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:52.217649 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 10:58:52.217683 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 10:58:52.218835 22408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:58:52.218912 22408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:58:52.218984 22402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:52.219671 22402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:52.225363 22402 catalog_manager.cc:1347] Generated new cluster ID: 4dbac5c47258430c8dbce45f7f9851f2
I20260927 10:58:52.225432 22402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:52.236606 22402 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:52.236910 22402 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:52.245314 22402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d95e433cc3c474ba0d0bdc37f170b4d: Generated new TSK 0
I20260927 10:58:52.246032 22402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:52.257400 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22315
2026-09-27T10:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:52 dist-test-slave-cffh krb5kdc[22414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:52 dist-test-slave-cffh krb5kdc[22414](info): set up 2 sockets
Sep 27 10:58:52 dist-test-slave-cffh krb5kdc[22414](info): commencing operation
krb5kdc: starting...
W20260927 10:58:53.278848 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:53 dist-test-slave-cffh krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:53.390801 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:53.495246 22427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:53.495458 22427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:53.495496 22427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:53.499511 22427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:53.499557 22427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:53.499575 22427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:53.499591 22427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:53.499641 22427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:53.504570 22427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:53.505415 22427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:53.506278 22427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:53.514097 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:53.514107 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:53.514235 22427 server_base.cc:1034] running on GCE node
W20260927 10:58:53.514107 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:53.514901 22427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:53.515877 22427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:53.517041 22427 hybrid_clock.cc:648] HybridClock initialized: now 1790506733517022 us; error 30 us; skew 500 ppm
Sep 27 10:58:53 dist-test-slave-cffh krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:53.519654 22427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:53.520502 22427 webserver.cc:460] Webserver started at http://127.0.0.1:37857/ using document root <none> and password file <none>
I20260927 10:58:53.521034 22427 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:53.521095 22427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:53.521308 22427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:53.523306 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "67b24bd8410d441ea6139b5ef99d4cbc"
format_stamp: "Formatted at 2026-09-27 10:58:53 on dist-test-slave-cffh"
server_key: "3ca40b1ca1b27b17f2e39c1a7530b60f"
server_key_iv: "a25a0f826d025a8b3525fecc470972e6"
server_key_version: "encryptionkey@0"
I20260927 10:58:53.523844 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "67b24bd8410d441ea6139b5ef99d4cbc"
format_stamp: "Formatted at 2026-09-27 10:58:53 on dist-test-slave-cffh"
server_key: "3ca40b1ca1b27b17f2e39c1a7530b60f"
server_key_iv: "a25a0f826d025a8b3525fecc470972e6"
server_key_version: "encryptionkey@0"
I20260927 10:58:53.527276 22427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:58:53.529469 22442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:53.530608 22427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:53.530753 22427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "67b24bd8410d441ea6139b5ef99d4cbc"
format_stamp: "Formatted at 2026-09-27 10:58:53 on dist-test-slave-cffh"
server_key: "3ca40b1ca1b27b17f2e39c1a7530b60f"
server_key_iv: "a25a0f826d025a8b3525fecc470972e6"
server_key_version: "encryptionkey@0"
I20260927 10:58:53.530853 22427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:53.555230 22427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:53.557986 22427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:53.558174 22427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:53.564913 22427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37059
I20260927 10:58:53.564954 22494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37059 every 8 connection(s)
I20260927 10:58:53.566103 22427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:53.568671 22495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:53.573629 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc: Bootstrap starting.
I20260927 10:58:53.575863 22495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:53.575867 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22427
I20260927 10:58:53.576005 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:53.576269 1211 external_mini_cluster.cc:1428] Setting key 168e21368b98513dd8c9b6305f1a9c25
I20260927 10:58:53.576617 22495 log.cc:824] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:53.578857 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc: No bootstrap required, opened a new log
I20260927 10:58:53.582939 22495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260927 10:58:53.583189 22495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:53.583258 22495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b24bd8410d441ea6139b5ef99d4cbc, State: Initialized, Role: FOLLOWER
I20260927 10:58:53.583730 22495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260927 10:58:53.583880 22495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:53.583968 22495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:53.584080 22495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:53.584900 22495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260927 10:58:53.585218 22495 leader_election.cc:302] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b24bd8410d441ea6139b5ef99d4cbc; no voters:
Sep 27 10:58:53 dist-test-slave-cffh krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:58:53.585520 22495 leader_election.cc:288] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:53.585623 22500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:53.585850 22500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [term 1 LEADER]: Becoming Leader. State: Replica: 67b24bd8410d441ea6139b5ef99d4cbc, State: Running, Role: LEADER
I20260927 10:58:53.586220 22500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260927 10:58:53.586807 22495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:53.587999 22501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } } }
I20260927 10:58:53.588124 22501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:53.588495 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b24bd8410d441ea6139b5ef99d4cbc. Latest consensus state: current_term: 1 leader_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b24bd8410d441ea6139b5ef99d4cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } } }
I20260927 10:58:53.589103 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:53.589527 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:53.591570 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:53.577351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:34447 (local address 127.0.0.1:37059)
0927 10:58:53.577752 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:53.577770 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:53.577810 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:58:53.578676 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:53.578698 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:53.579091 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:53.579456 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:53.580180 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:53.581679 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:53.582556 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:53.583267 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:53.583448 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:53.586292 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:53.586298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:53.586313 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:53.586343 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:53.588590 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:53.589179 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:53.589199 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:53.589206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:53.589337 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:53.589715 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:53.589718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:53.589719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:53.590072 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:53.590226 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:53.590556 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 10:58:53.590875 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260927 10:58:53.592300 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:53.594254 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:53.597828 22509 catalog_manager.cc:1347] Generated new cluster ID: 79052d59762b49a189a075c6fef5fa57
I20260927 10:58:53.597886 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:53.605912 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:53.596223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:42919 (local address 127.0.0.1:37059)
0927 10:58:53.596335 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:53.596339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:53.596693 (+ 354us) server_negotiation.cc:407] Connection header received
0927 10:58:53.596848 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:53.596851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:53.596896 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:53.596971 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:53.598659 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:53.599471 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:53.600576 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:53.600930 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:53.601034 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:53.602522 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:53.602527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:53.602531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:53.602557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:53.604186 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:53.604688 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:53.604692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:53.604694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:53.604776 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:53.605088 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:53.605091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:53.605093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:53.605289 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:53.605442 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:53.605677 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 10:58:53.605768 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:58:53.610543 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:53.611410 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:53.621268 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b24bd8410d441ea6139b5ef99d4cbc: Generated new TSK 0
I20260927 10:58:53.621846 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:53.628983 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22427
W20260927 10:58:53.634153 22412 connection.cc:535] client connection to 127.0.0.1:37059 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37059): Connection reset by peer (error 104)
2026-09-27T10:58:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:53 dist-test-slave-cffh krb5kdc[22526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:53 dist-test-slave-cffh krb5kdc[22526](info): set up 2 sockets
Sep 27 10:58:53 dist-test-slave-cffh krb5kdc[22526](info): commencing operation
krb5kdc: starting...
W20260927 10:58:54.662868 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:54 dist-test-slave-cffh krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:54.778648 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:54.885156 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:54.885390 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:54.885427 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:54.889279 22539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:54.889326 22539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:54.889345 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:54.889361 22539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:54.889401 22539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:54.894484 22539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:54.895483 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:54.896440 22539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:54.904551 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:54.904543 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:54.904543 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:54.904645 22539 server_base.cc:1034] running on GCE node
I20260927 10:58:54.905289 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:54.906291 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:54.907464 22539 hybrid_clock.cc:648] HybridClock initialized: now 1790506734907410 us; error 62 us; skew 500 ppm
Sep 27 10:58:54 dist-test-slave-cffh krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:54.910171 22539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:54.910977 22539 webserver.cc:460] Webserver started at http://127.0.0.1:39535/ using document root <none> and password file <none>
I20260927 10:58:54.911475 22539 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:54.911532 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:54.911686 22539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:54.913579 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "42fcbe99141944caa19eabfaaad73b1d"
format_stamp: "Formatted at 2026-09-27 10:58:54 on dist-test-slave-cffh"
server_key: "9d9b98303d6394ba138470a94029d84d"
server_key_iv: "2ce2b07197b7bece7114e25166649cff"
server_key_version: "encryptionkey@0"
I20260927 10:58:54.914076 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "42fcbe99141944caa19eabfaaad73b1d"
format_stamp: "Formatted at 2026-09-27 10:58:54 on dist-test-slave-cffh"
server_key: "9d9b98303d6394ba138470a94029d84d"
server_key_iv: "2ce2b07197b7bece7114e25166649cff"
server_key_version: "encryptionkey@0"
I20260927 10:58:54.917505 22539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:58:54.919747 22554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:54.920920 22539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:58:54.921036 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "42fcbe99141944caa19eabfaaad73b1d"
format_stamp: "Formatted at 2026-09-27 10:58:54 on dist-test-slave-cffh"
server_key: "9d9b98303d6394ba138470a94029d84d"
server_key_iv: "2ce2b07197b7bece7114e25166649cff"
server_key_version: "encryptionkey@0"
I20260927 10:58:54.921129 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:54.937659 22539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:54.940246 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:54.940430 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:54.947208 22539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33691
I20260927 10:58:54.947278 22606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33691 every 8 connection(s)
I20260927 10:58:54.948127 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:54.950850 22607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:54.954044 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22539
I20260927 10:58:54.954216 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:54.954506 1211 external_mini_cluster.cc:1428] Setting key b7b1b21a1749be9039ae5a836a03f267
I20260927 10:58:54.956331 22607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d: Bootstrap starting.
I20260927 10:58:54.958514 22607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:54.959313 22607 log.cc:824] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:54.961207 22607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d: No bootstrap required, opened a new log
I20260927 10:58:54.963256 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:54.955862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:43971 (local address 127.0.0.1:33691)
0927 10:58:54.956252 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:54.956268 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:54.956308 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:58:54.956965 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:54.956981 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:54.957300 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:54.957592 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:54.957911 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:54.958955 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:54.960053 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:54.960506 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:54.960626 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:54.961472 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:54.961477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:54.961488 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:54.961855 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:54.961903 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:54.961924 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:54.962024 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:54.962204 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:58:54.962341 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":121,"threads_started":1}
I20260927 10:58:54.964284 22607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbe99141944caa19eabfaaad73b1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260927 10:58:54.964524 22607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:54.964576 22607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42fcbe99141944caa19eabfaaad73b1d, State: Initialized, Role: FOLLOWER
I20260927 10:58:54.964967 22607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbe99141944caa19eabfaaad73b1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260927 10:58:54.965119 22607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:54.965183 22607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:54.965282 22607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:54.966024 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:54.966293 22607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbe99141944caa19eabfaaad73b1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260927 10:58:54.966645 22607 leader_election.cc:302] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42fcbe99141944caa19eabfaaad73b1d; no voters:
I20260927 10:58:54.966944 22607 leader_election.cc:288] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:54.967005 22612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:54.967226 22612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [term 1 LEADER]: Becoming Leader. State: Replica: 42fcbe99141944caa19eabfaaad73b1d, State: Running, Role: LEADER
I20260927 10:58:54.967550 22612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbe99141944caa19eabfaaad73b1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260927 10:58:54.968106 22607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:54.968649 22612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42fcbe99141944caa19eabfaaad73b1d. Latest consensus state: current_term: 1 leader_uuid: "42fcbe99141944caa19eabfaaad73b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbe99141944caa19eabfaaad73b1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } } }
I20260927 10:58:54.968878 22612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:54.969063 22616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42fcbe99141944caa19eabfaaad73b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbe99141944caa19eabfaaad73b1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } } }
I20260927 10:58:54.969172 22616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:54.969291 22624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:54.972379 22624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:54.974864 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:54.970549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:36391 (local address 127.0.0.1:33691)
0927 10:58:54.970673 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:54.970677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:54.970852 (+ 175us) server_negotiation.cc:407] Connection header received
0927 10:58:54.971016 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:54.971020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:54.971093 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:54.971163 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:54.971481 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:54.971978 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:54.973185 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:54.973418 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:54.973520 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:54.974326 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:54.974330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:54.974332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:54.974344 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:54.974368 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:54.974380 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:54.974436 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:54.974711 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 10:58:54.974742 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:58:54.977813 22624 catalog_manager.cc:1347] Generated new cluster ID: 589c21e86c9e4906b679eecb782c2e8f
I20260927 10:58:54.977875 22624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:54.985505 22624 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:54.986306 22624 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:54.990227 22624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42fcbe99141944caa19eabfaaad73b1d: Generated new TSK 0
I20260927 10:58:54.990816 22624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:54.995589 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22539
W20260927 10:58:55.000806 22524 connection.cc:535] client connection to 127.0.0.1:33691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33691): Connection reset by peer (error 104)
2026-09-27T10:58:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:55 dist-test-slave-cffh krb5kdc[22638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:55 dist-test-slave-cffh krb5kdc[22638](info): set up 2 sockets
Sep 27 10:58:55 dist-test-slave-cffh krb5kdc[22638](info): commencing operation
krb5kdc: starting...
W20260927 10:58:56.010839 1211 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:56 dist-test-slave-cffh krb5kdc[22638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:58:56.123071 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:56.230522 22651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:56.230826 22651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:56.230865 22651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:56.234655 22651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:56.234697 22651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:56.234715 22651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:56.234731 22651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:56.234745 22651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:56.239646 22651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:56.240496 22651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:56.241358 22651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:56.249366 22659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:56.249400 22651 server_base.cc:1034] running on GCE node
W20260927 10:58:56.249369 22656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:56.249366 22657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:56.250137 22651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:56.251214 22651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:56.252408 22651 hybrid_clock.cc:648] HybridClock initialized: now 1790506736252393 us; error 24 us; skew 500 ppm
Sep 27 10:58:56 dist-test-slave-cffh krb5kdc[22638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:56.260166 22651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:58:56.261086 22651 webserver.cc:460] Webserver started at http://127.0.0.1:46331/ using document root <none> and password file <none>
I20260927 10:58:56.261641 22651 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:56.261772 22651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:56.261986 22651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:56.264030 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "60a4a92fed7641c3b21effd4f70e5ee2"
format_stamp: "Formatted at 2026-09-27 10:58:56 on dist-test-slave-cffh"
server_key: "5672974d9472cb5bda0296ecc16f16bf"
server_key_iv: "e0016b79aa4c40cbc074767dc5f21e77"
server_key_version: "encryptionkey@0"
I20260927 10:58:56.264557 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "60a4a92fed7641c3b21effd4f70e5ee2"
format_stamp: "Formatted at 2026-09-27 10:58:56 on dist-test-slave-cffh"
server_key: "5672974d9472cb5bda0296ecc16f16bf"
server_key_iv: "e0016b79aa4c40cbc074767dc5f21e77"
server_key_version: "encryptionkey@0"
I20260927 10:58:56.268092 22651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:58:56.270375 22666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:56.271952 22651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:58:56.272132 22651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "60a4a92fed7641c3b21effd4f70e5ee2"
format_stamp: "Formatted at 2026-09-27 10:58:56 on dist-test-slave-cffh"
server_key: "5672974d9472cb5bda0296ecc16f16bf"
server_key_iv: "e0016b79aa4c40cbc074767dc5f21e77"
server_key_version: "encryptionkey@0"
I20260927 10:58:56.272282 22651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:56.277297 22651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:56.279918 22651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:56.280049 22651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:56.286288 22651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38861
I20260927 10:58:56.286300 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38861 every 8 connection(s)
I20260927 10:58:56.287170 22651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:56.288097 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22651
I20260927 10:58:56.288240 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:56.288543 1211 external_mini_cluster.cc:1428] Setting key 7c58bd67be58e171f028bcc6eb453c95
I20260927 10:58:56.290112 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:56.293396 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:56.289825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:59155 (local address 127.0.0.1:38861)
0927 10:58:56.290295 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:56.290310 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:56.290343 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:58:56.291023 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:56.291040 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:56.291298 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:56.291607 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:56.292325 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:56.292336 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:56.292347 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:56.292357 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:56.292403 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:56.292420 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:56.292550 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:56.292626 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:58:56.292783 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260927 10:58:56.295954 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2: Bootstrap starting.
I20260927 10:58:56.296030 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:56.298065 22719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:56.298969 22719 log.cc:824] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:56.299582 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:56.297906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:55273 (local address 127.0.0.1:38861)
0927 10:58:56.298033 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:56.298038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:56.298206 (+ 168us) server_negotiation.cc:407] Connection header received
0927 10:58:56.298285 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:56.298289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:56.298325 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:56.298408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:56.299040 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:56.299043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:56.299046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:56.299059 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:56.299085 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:56.299100 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:56.299182 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:56.299378 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 10:58:56.299411 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:58:56.301155 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2: No bootstrap required, opened a new log
I20260927 10:58:56.304565 22719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260927 10:58:56.304790 22719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:56.304850 22719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a4a92fed7641c3b21effd4f70e5ee2, State: Initialized, Role: FOLLOWER
I20260927 10:58:56.305199 22719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260927 10:58:56.305331 22719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:56.305374 22719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:56.305440 22719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:56.306382 22719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260927 10:58:56.306738 22719 leader_election.cc:302] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60a4a92fed7641c3b21effd4f70e5ee2; no voters:
I20260927 10:58:56.307060 22719 leader_election.cc:288] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:56.307159 22729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:56.307395 22729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 60a4a92fed7641c3b21effd4f70e5ee2, State: Running, Role: LEADER
I20260927 10:58:56.307698 22729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260927 10:58:56.308198 22719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:56.309296 22731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60a4a92fed7641c3b21effd4f70e5ee2. Latest consensus state: current_term: 1 leader_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } } }
I20260927 10:58:56.309427 22731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:56.309747 22730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a4a92fed7641c3b21effd4f70e5ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } } }
I20260927 10:58:56.309909 22730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:56.310117 22738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:56.312767 22738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:56.317727 22738 catalog_manager.cc:1347] Generated new cluster ID: d9ccd65d656944d7be950caf36f243dd
I20260927 10:58:56.317827 22738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:56.331236 22738 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:56.331547 22738 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:56.338701 22738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60a4a92fed7641c3b21effd4f70e5ee2: Generated new TSK 0
I20260927 10:58:56.339236 22738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:56.345677 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22651
2026-09-27T10:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:56 dist-test-slave-cffh krb5kdc[22750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:56 dist-test-slave-cffh krb5kdc[22750](info): set up 2 sockets
Sep 27 10:58:56 dist-test-slave-cffh krb5kdc[22750](info): commencing operation
krb5kdc: starting...
W20260927 10:58:57.375097 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:57 dist-test-slave-cffh krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:58:57.487556 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40305
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:57.593286 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:57.593511 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:57.593549 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:57.597429 22763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:57.597478 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:57.597496 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:57.597512 22763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:57.597525 22763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:57.602478 22763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:57.603360 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:57.604238 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:57.612262 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:57.612450 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:57.612289 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:57.612521 22763 server_base.cc:1034] running on GCE node
I20260927 10:58:57.613114 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:57.614094 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:57.615283 22763 hybrid_clock.cc:648] HybridClock initialized: now 1790506737615263 us; error 28 us; skew 500 ppm
Sep 27 10:58:57 dist-test-slave-cffh krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:57.617967 22763 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:58:57.618821 22763 webserver.cc:460] Webserver started at http://127.1.46.254:34363/ using document root <none> and password file <none>
I20260927 10:58:57.619320 22763 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:57.619375 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:57.619526 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:57.621362 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "e958bbefc40c4904b1e646adf14ac4ab"
format_stamp: "Formatted at 2026-09-27 10:58:57 on dist-test-slave-cffh"
server_key: "eb84609186a987b78513e5015e2020bc"
server_key_iv: "34470c91ed0c8461963bc5cb1a3259ef"
server_key_version: "encryptionkey@0"
I20260927 10:58:57.621906 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "e958bbefc40c4904b1e646adf14ac4ab"
format_stamp: "Formatted at 2026-09-27 10:58:57 on dist-test-slave-cffh"
server_key: "eb84609186a987b78513e5015e2020bc"
server_key_iv: "34470c91ed0c8461963bc5cb1a3259ef"
server_key_version: "encryptionkey@0"
I20260927 10:58:57.625332 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:58:57.627677 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:57.628906 22763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:58:57.629006 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "e958bbefc40c4904b1e646adf14ac4ab"
format_stamp: "Formatted at 2026-09-27 10:58:57 on dist-test-slave-cffh"
server_key: "eb84609186a987b78513e5015e2020bc"
server_key_iv: "34470c91ed0c8461963bc5cb1a3259ef"
server_key_version: "encryptionkey@0"
I20260927 10:58:57.629104 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:57.644744 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:57.647895 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:57.648043 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:57.654958 22763 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:40305
I20260927 10:58:57.654978 22830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:40305 every 8 connection(s)
I20260927 10:58:57.655881 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:57.658553 22831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:57.662420 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22763
I20260927 10:58:57.662557 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:57.662859 1211 external_mini_cluster.cc:1428] Setting key c1ae4abbac83ad9daf39cf2b740a0a96
I20260927 10:58:57.664196 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab: Bootstrap starting.
I20260927 10:58:57.666896 22831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:57.667794 22831 log.cc:824] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:57.669781 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab: No bootstrap required, opened a new log
Sep 27 10:58:57 dist-test-slave-cffh krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:58:57.673622 22831 raft_consensus.cc:348] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e958bbefc40c4904b1e646adf14ac4ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40305 } }
I20260927 10:58:57.673908 22831 raft_consensus.cc:374] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:57.673971 22831 raft_consensus.cc:729] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e958bbefc40c4904b1e646adf14ac4ab, State: Initialized, Role: FOLLOWER
I20260927 10:58:57.674470 22831 consensus_queue.cc:260] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e958bbefc40c4904b1e646adf14ac4ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40305 } }
I20260927 10:58:57.674643 22831 raft_consensus.cc:388] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:57.674729 22831 raft_consensus.cc:482] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:57.674829 22831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:57.675894 22831 raft_consensus.cc:504] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e958bbefc40c4904b1e646adf14ac4ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40305 } }
I20260927 10:58:57.676270 22831 leader_election.cc:302] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e958bbefc40c4904b1e646adf14ac4ab; no voters:
I20260927 10:58:57.676630 22831 leader_election.cc:288] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:57.676805 22836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:57.677052 22836 raft_consensus.cc:686] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [term 1 LEADER]: Becoming Leader. State: Replica: e958bbefc40c4904b1e646adf14ac4ab, State: Running, Role: LEADER
I20260927 10:58:57.677657 22836 consensus_queue.cc:237] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e958bbefc40c4904b1e646adf14ac4ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40305 } }
I20260927 10:58:57.677934 22831 sys_catalog.cc:564] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:57.678820 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e958bbefc40c4904b1e646adf14ac4ab. Latest consensus state: current_term: 1 leader_uuid: "e958bbefc40c4904b1e646adf14ac4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e958bbefc40c4904b1e646adf14ac4ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40305 } } }
I20260927 10:58:57.678932 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:57.678973 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:57.664247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43967 (local address 127.1.46.254:40305)
0927 10:58:57.664679 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:57.664697 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:57.664737 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:58:57.665619 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:57.665641 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:57.666063 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:57.666411 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:57.667196 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:57.668548 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:57.669459 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:57.670148 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:57.670304 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:57.672906 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:57.672912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:57.672928 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:57.672958 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:57.675162 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:57.675654 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:57.675659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:57.675664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:57.675824 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:57.676177 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:57.676180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:57.676182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:57.676805 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:57.677594 (+ 789us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:57.677768 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 10:58:57.677980 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260927 10:58:57.679315 22838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e958bbefc40c4904b1e646adf14ac4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e958bbefc40c4904b1e646adf14ac4ab" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40305 } } }
I20260927 10:58:57.679512 22838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:57.682367 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 10:58:57.683096 22851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:58:57.683207 22851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:58:57.684162 22845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:57.685103 22845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:57.690630 22845 catalog_manager.cc:1347] Generated new cluster ID: c5629900eda4402b9b9155bace02a9c6
I20260927 10:58:57.690680 22845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:57.697070 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:57.684695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56541 (local address 127.1.46.254:40305)
0927 10:58:57.684799 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:57.684803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:57.684815 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:58:57.684848 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:57.684851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:57.684888 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:57.684970 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:57.685818 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:57.686510 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:57.687633 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:57.687957 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:57.688065 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:57.689694 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:57.689697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:57.689700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:58:57.689723 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:57.695325 (+ 5602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:57.695855 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:57.695858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:57.695860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:57.695937 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:58:57.696278 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:58:57.696281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:58:57.696283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:58:57.696456 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:57.696579 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:57.696850 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:58:57.696935 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:58:57.704346 22845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:57.705718 22845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:57.715479 22845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e958bbefc40c4904b1e646adf14ac4ab: Generated new TSK 0
I20260927 10:58:57.716116 22845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:57.722330 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22763
W20260927 10:58:57.728806 22748 connection.cc:535] client connection to 127.1.46.254:40305 recv error: Network error: failed to read from TLS socket (remote: 127.1.46.254:40305): Connection reset by peer (error 104)
2026-09-27T10:58:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:57 dist-test-slave-cffh krb5kdc[22862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:57 dist-test-slave-cffh krb5kdc[22862](info): set up 2 sockets
Sep 27 10:58:57 dist-test-slave-cffh krb5kdc[22862](info): commencing operation
krb5kdc: starting...
W20260927 10:58:58.758518 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:58:58 dist-test-slave-cffh krb5kdc[22862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:58:58.870783 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:38595
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:58:58.975009 22875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:58:58.975256 22875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:58:58.975320 22875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:58:58.979210 22875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:58:58.979274 22875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:58:58.979326 22875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:58:58.979374 22875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:58:58.979413 22875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:58:58.984239 22875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:38595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:58:58.985091 22875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:58:58.986006 22875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:58:58.993659 22883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:58.993829 22880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:58:58.993813 22881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:58:58.994087 22875 server_base.cc:1034] running on GCE node
I20260927 10:58:58.994664 22875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:58:58.995678 22875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:58:58.996888 22875 hybrid_clock.cc:648] HybridClock initialized: now 1790506738996865 us; error 41 us; skew 500 ppm
Sep 27 10:58:58 dist-test-slave-cffh krb5kdc[22862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:58:58.999683 22875 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:58:59.000465 22875 webserver.cc:460] Webserver started at http://127.1.46.254:45231/ using document root <none> and password file <none>
I20260927 10:58:59.000967 22875 fs_manager.cc:359] Metadata directory not provided
I20260927 10:58:59.001024 22875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:58:59.001242 22875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:58:59.003157 22875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "f7f23d70d2374dc99025412077644402"
format_stamp: "Formatted at 2026-09-27 10:58:58 on dist-test-slave-cffh"
server_key: "1aed78a387757b6704aeeda0d1de025e"
server_key_iv: "f10725b1fd77b17a8eb5fb8205183319"
server_key_version: "encryptionkey@0"
I20260927 10:58:59.003680 22875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "f7f23d70d2374dc99025412077644402"
format_stamp: "Formatted at 2026-09-27 10:58:58 on dist-test-slave-cffh"
server_key: "1aed78a387757b6704aeeda0d1de025e"
server_key_iv: "f10725b1fd77b17a8eb5fb8205183319"
server_key_version: "encryptionkey@0"
I20260927 10:58:59.007041 22875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:58:59.009336 22890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:58:59.010396 22875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:58:59.010510 22875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "f7f23d70d2374dc99025412077644402"
format_stamp: "Formatted at 2026-09-27 10:58:58 on dist-test-slave-cffh"
server_key: "1aed78a387757b6704aeeda0d1de025e"
server_key_iv: "f10725b1fd77b17a8eb5fb8205183319"
server_key_version: "encryptionkey@0"
I20260927 10:58:59.010654 22875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:58:59.025471 22875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:58:59.028046 22875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:58:59.028244 22875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:58:59.034614 22875 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:38595
I20260927 10:58:59.034704 22942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:38595 every 8 connection(s)
I20260927 10:58:59.035470 22875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:58:59.036419 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22875
I20260927 10:58:59.036557 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:58:59.036821 1211 external_mini_cluster.cc:1428] Setting key 30c75289ad5f514d2e84c78afbf42874
I20260927 10:58:59.038200 22943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:58:59.043916 22943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402: Bootstrap starting.
I20260927 10:58:59.045034 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:59.038085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60205 (local address 127.1.46.254:38595)
0927 10:58:59.038457 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:59.038471 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:59.038503 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:58:59.039112 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:59.039126 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:59.039427 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:59.039684 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:59.040048 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:59.040967 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:59.042127 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:59.042623 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:59.042749 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:59.043538 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:59.043542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:59.043554 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:59.043895 (+ 341us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:59.043943 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:59.043962 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:59.044108 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:59.044254 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 10:58:59.044396 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":125,"threads_started":1}
I20260927 10:58:59.046120 22943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402: Neither blocks nor log segments found. Creating new log.
I20260927 10:58:59.046888 22943 log.cc:824] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402: Log is configured to *not* fsync() on all Append() calls
I20260927 10:58:59.047785 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:58:59.049305 22943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402: No bootstrap required, opened a new log
I20260927 10:58:59.053529 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:58:59.049732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39023 (local address 127.1.46.254:38595)
0927 10:58:59.049833 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:58:59.049837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:58:59.049990 (+ 153us) server_negotiation.cc:407] Connection header received
0927 10:58:59.050095 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:58:59.050097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:58:59.050144 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:58:59.050202 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:58:59.050454 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:59.050969 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:59.052038 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:58:59.052242 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:58:59.052334 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:58:59.053072 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:58:59.053075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:58:59.053077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:58:59.053086 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:58:59.053106 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:58:59.053115 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:58:59.053177 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:58:59.053342 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 10:58:59.053387 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:58:59.053576 22943 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f23d70d2374dc99025412077644402" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38595 } }
I20260927 10:58:59.053845 22943 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:58:59.053903 22943 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f23d70d2374dc99025412077644402, State: Initialized, Role: FOLLOWER
I20260927 10:58:59.054270 22943 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f23d70d2374dc99025412077644402" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38595 } }
I20260927 10:58:59.054420 22943 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:58:59.054507 22943 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:58:59.054629 22943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:58:59.055397 22943 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f23d70d2374dc99025412077644402" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38595 } }
I20260927 10:58:59.055648 22943 leader_election.cc:302] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f23d70d2374dc99025412077644402; no voters:
I20260927 10:58:59.055866 22943 leader_election.cc:288] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:58:59.055941 22953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:58:59.056214 22953 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [term 1 LEADER]: Becoming Leader. State: Replica: f7f23d70d2374dc99025412077644402, State: Running, Role: LEADER
I20260927 10:58:59.056538 22953 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f23d70d2374dc99025412077644402" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38595 } }
I20260927 10:58:59.056979 22943 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:58:59.058123 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f23d70d2374dc99025412077644402. Latest consensus state: current_term: 1 leader_uuid: "f7f23d70d2374dc99025412077644402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f23d70d2374dc99025412077644402" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38595 } } }
I20260927 10:58:59.058125 22954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f23d70d2374dc99025412077644402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f23d70d2374dc99025412077644402" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 38595 } } }
I20260927 10:58:59.058321 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:59.058342 22954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402 [sys.catalog]: This master's current role is: LEADER
I20260927 10:58:59.058789 22963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:58:59.061998 22963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:58:59.066612 22963 catalog_manager.cc:1347] Generated new cluster ID: 1760665de81e4842b796e952d2b8e7a4
I20260927 10:58:59.066677 22963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:58:59.093037 22963 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:58:59.094029 22963 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:58:59.099154 22963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7f23d70d2374dc99025412077644402: Generated new TSK 0
I20260927 10:58:59.099951 22963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:58:59.105353 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22875
W20260927 10:58:59.110646 22860 connection.cc:535] client connection to 127.1.46.254:38595 recv error: Network error: failed to read from TLS socket (remote: 127.1.46.254:38595): Connection reset by peer (error 104)
2026-09-27T10:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:58:59 dist-test-slave-cffh krb5kdc[22974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:58:59 dist-test-slave-cffh krb5kdc[22974](info): set up 2 sockets
Sep 27 10:58:59 dist-test-slave-cffh krb5kdc[22974](info): commencing operation
krb5kdc: starting...
W20260927 10:59:00.178895 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:59:00 dist-test-slave-cffh krb5kdc[22974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:59:00.293929 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:37531
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:37531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:59:00.406736 22987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:00.406980 22987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:00.407061 22987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:00.411103 22987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:00.411186 22987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:00.411242 22987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:00.411274 22987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:00.411361 22987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:00.416564 22987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:37531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:37531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:00.417479 22987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:00.418426 22987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:59:00.426985 22992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:00.426996 22993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:00.427055 22987 server_base.cc:1034] running on GCE node
W20260927 10:59:00.426996 22995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:00.427700 22987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:00.428712 22987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:00.429868 22987 hybrid_clock.cc:648] HybridClock initialized: now 1790506740429857 us; error 26 us; skew 500 ppm
Sep 27 10:59:00 dist-test-slave-cffh krb5kdc[22974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506740, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:00.432487 22987 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:59:00.433213 22987 webserver.cc:460] Webserver started at http://127.1.46.254:34239/ using document root <none> and password file <none>
I20260927 10:59:00.433686 22987 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:00.433739 22987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:00.433888 22987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:00.435775 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "b06a96c1601e4bc1be1874c061d7beb1"
format_stamp: "Formatted at 2026-09-27 10:59:00 on dist-test-slave-cffh"
server_key: "bf86abf43bf11fcb0206bf48e89d96a3"
server_key_iv: "a595755ce7e45c751722d21f2d59813c"
server_key_version: "encryptionkey@0"
I20260927 10:59:00.436245 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "b06a96c1601e4bc1be1874c061d7beb1"
format_stamp: "Formatted at 2026-09-27 10:59:00 on dist-test-slave-cffh"
server_key: "bf86abf43bf11fcb0206bf48e89d96a3"
server_key_iv: "a595755ce7e45c751722d21f2d59813c"
server_key_version: "encryptionkey@0"
I20260927 10:59:00.439628 22987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:59:00.441723 23002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:00.442896 22987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:59:00.443028 22987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "b06a96c1601e4bc1be1874c061d7beb1"
format_stamp: "Formatted at 2026-09-27 10:59:00 on dist-test-slave-cffh"
server_key: "bf86abf43bf11fcb0206bf48e89d96a3"
server_key_iv: "a595755ce7e45c751722d21f2d59813c"
server_key_version: "encryptionkey@0"
I20260927 10:59:00.443130 22987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:00.447252 22987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:00.449813 22987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:00.449985 22987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:00.456436 22987 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:37531
I20260927 10:59:00.456490 23054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:37531 every 8 connection(s)
I20260927 10:59:00.457265 22987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:59:00.459314 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 22987
I20260927 10:59:00.459467 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:59:00.459721 1211 external_mini_cluster.cc:1428] Setting key 95ac81de11db35e1282c9562c2b7bc89
I20260927 10:59:00.459872 23055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:00.464859 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:00.460990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46773 (local address 127.1.46.254:37531)
0927 10:59:00.461369 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:00.461385 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:00.461421 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:59:00.462327 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:00.462349 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:00.462706 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:00.463041 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:00.463636 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:00.463641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:00.463654 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:59:00.463665 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:00.463716 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:59:00.463740 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:00.463868 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:00.463931 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 10:59:00.464072 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260927 10:59:00.466507 23055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1: Bootstrap starting.
I20260927 10:59:00.467442 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:59:00.468722 23055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1: Neither blocks nor log segments found. Creating new log.
I20260927 10:59:00.469439 23055 log.cc:824] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:00.470938 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:00.469381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36533 (local address 127.1.46.254:37531)
0927 10:59:00.469500 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:00.469504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:00.469701 (+ 197us) server_negotiation.cc:407] Connection header received
0927 10:59:00.469854 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:00.469856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:00.469889 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:00.469976 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:00.470536 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:00.470539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:00.470540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:59:00.470549 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:00.470608 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:59:00.470625 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:00.470690 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:00.470792 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 10:59:00.470823 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:59:00.471602 23055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1: No bootstrap required, opened a new log
I20260927 10:59:00.474822 23055 raft_consensus.cc:348] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06a96c1601e4bc1be1874c061d7beb1" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37531 } }
I20260927 10:59:00.475010 23055 raft_consensus.cc:374] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:59:00.475080 23055 raft_consensus.cc:729] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06a96c1601e4bc1be1874c061d7beb1, State: Initialized, Role: FOLLOWER
I20260927 10:59:00.475504 23055 consensus_queue.cc:260] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06a96c1601e4bc1be1874c061d7beb1" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37531 } }
I20260927 10:59:00.475642 23055 raft_consensus.cc:388] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:59:00.475708 23055 raft_consensus.cc:482] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:59:00.475826 23055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:00.476845 23055 raft_consensus.cc:504] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06a96c1601e4bc1be1874c061d7beb1" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37531 } }
I20260927 10:59:00.477140 23055 leader_election.cc:302] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06a96c1601e4bc1be1874c061d7beb1; no voters:
I20260927 10:59:00.477401 23055 leader_election.cc:288] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:59:00.477535 23065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:59:00.477758 23065 raft_consensus.cc:686] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [term 1 LEADER]: Becoming Leader. State: Replica: b06a96c1601e4bc1be1874c061d7beb1, State: Running, Role: LEADER
I20260927 10:59:00.478080 23065 consensus_queue.cc:237] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06a96c1601e4bc1be1874c061d7beb1" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37531 } }
I20260927 10:59:00.478559 23055 sys_catalog.cc:564] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:00.479763 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b06a96c1601e4bc1be1874c061d7beb1. Latest consensus state: current_term: 1 leader_uuid: "b06a96c1601e4bc1be1874c061d7beb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06a96c1601e4bc1be1874c061d7beb1" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37531 } } }
I20260927 10:59:00.479885 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:00.480350 23074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:59:00.480484 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06a96c1601e4bc1be1874c061d7beb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06a96c1601e4bc1be1874c061d7beb1" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 37531 } } }
I20260927 10:59:00.480598 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:00.483428 23074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:59:00.488574 23074 catalog_manager.cc:1347] Generated new cluster ID: 910eab860d464fa7a87736af98d50ad8
I20260927 10:59:00.488632 23074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:59:00.499585 23074 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:59:00.499855 23074 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:59:00.507745 23074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b06a96c1601e4bc1be1874c061d7beb1: Generated new TSK 0
I20260927 10:59:00.508286 23074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:59:00.515722 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 22987
2026-09-27T10:59:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:59:00 dist-test-slave-cffh krb5kdc[23086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:59:00 dist-test-slave-cffh krb5kdc[23086](info): set up 2 sockets
Sep 27 10:59:00 dist-test-slave-cffh krb5kdc[23086](info): commencing operation
krb5kdc: starting...
W20260927 10:59:01.538786 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:59:01 dist-test-slave-cffh krb5kdc[23086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:59:01.655076 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44693
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:44693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:59:01.760495 23099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:01.760721 23099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:01.760814 23099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:01.764684 23099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:01.764731 23099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:01.764779 23099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:01.764819 23099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:01.764855 23099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:01.769848 23099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:44693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:01.770817 23099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:01.771755 23099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:59:01.779863 23107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:01.779920 23104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:01.779953 23105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:01.780084 23099 server_base.cc:1034] running on GCE node
I20260927 10:59:01.780686 23099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:01.781694 23099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:01.782857 23099 hybrid_clock.cc:648] HybridClock initialized: now 1790506741782844 us; error 25 us; skew 500 ppm
Sep 27 10:59:01 dist-test-slave-cffh krb5kdc[23086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506741, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:01.790638 23099 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:59:01.791637 23099 webserver.cc:460] Webserver started at http://127.1.46.254:33753/ using document root <none> and password file <none>
I20260927 10:59:01.792225 23099 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:01.792311 23099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:01.792554 23099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:01.794637 23099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "8177adde9826459e95bfd8a182a09a99"
format_stamp: "Formatted at 2026-09-27 10:59:01 on dist-test-slave-cffh"
server_key: "f5542bbbd1e7f6274b5367b51e589adc"
server_key_iv: "3bc94ffbf147916ff5597fb2eee0e7a7"
server_key_version: "encryptionkey@0"
I20260927 10:59:01.795176 23099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "8177adde9826459e95bfd8a182a09a99"
format_stamp: "Formatted at 2026-09-27 10:59:01 on dist-test-slave-cffh"
server_key: "f5542bbbd1e7f6274b5367b51e589adc"
server_key_iv: "3bc94ffbf147916ff5597fb2eee0e7a7"
server_key_version: "encryptionkey@0"
I20260927 10:59:01.798729 23099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:59:01.801018 23114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:01.802217 23099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:59:01.802345 23099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8177adde9826459e95bfd8a182a09a99"
format_stamp: "Formatted at 2026-09-27 10:59:01 on dist-test-slave-cffh"
server_key: "f5542bbbd1e7f6274b5367b51e589adc"
server_key_iv: "3bc94ffbf147916ff5597fb2eee0e7a7"
server_key_version: "encryptionkey@0"
I20260927 10:59:01.802446 23099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:01.827674 23099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:01.830876 23099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:01.831063 23099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:01.837958 23099 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:44693
I20260927 10:59:01.837981 23166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:44693 every 8 connection(s)
I20260927 10:59:01.838850 23099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:59:01.840286 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23099
I20260927 10:59:01.840432 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:59:01.840695 1211 external_mini_cluster.cc:1428] Setting key df7e0191fbcddc0d61794d9f3472b0f6
I20260927 10:59:01.841646 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:01.848012 23167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99: Bootstrap starting.
Sep 27 10:59:01 dist-test-slave-cffh krb5kdc[23086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:59:01.850328 23167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99: Neither blocks nor log segments found. Creating new log.
I20260927 10:59:01.851553 23167 log.cc:824] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:01.853658 23167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99: No bootstrap required, opened a new log
I20260927 10:59:01.855486 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:01.842009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49867 (local address 127.1.46.254:44693)
0927 10:59:01.842404 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:01.842422 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:01.842460 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:59:01.843340 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:01.843361 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:01.843771 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:01.844132 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:01.844751 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:01.846146 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:01.846990 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:01.847649 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:01.847808 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:01.850657 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:01.850692 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:01.850711 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:01.850776 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:01.852690 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:01.853154 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:01.853158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:01.853162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:01.853259 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:01.853615 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:01.853618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:01.853619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:01.854050 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:01.854197 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:01.854478 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 10:59:01.854823 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260927 10:59:01.857551 23167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8177adde9826459e95bfd8a182a09a99" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44693 } }
I20260927 10:59:01.857823 23167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:59:01.857914 23167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8177adde9826459e95bfd8a182a09a99, State: Initialized, Role: FOLLOWER
I20260927 10:59:01.858323 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:59:01.858379 23167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8177adde9826459e95bfd8a182a09a99" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44693 } }
I20260927 10:59:01.858538 23167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:59:01.858624 23167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:59:01.858716 23167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:01.859814 23167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8177adde9826459e95bfd8a182a09a99" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44693 } }
I20260927 10:59:01.860137 23167 leader_election.cc:302] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8177adde9826459e95bfd8a182a09a99; no voters:
I20260927 10:59:01.860472 23167 leader_election.cc:288] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:59:01.860554 23176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:59:01.860854 23176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [term 1 LEADER]: Becoming Leader. State: Replica: 8177adde9826459e95bfd8a182a09a99, State: Running, Role: LEADER
I20260927 10:59:01.861160 23176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8177adde9826459e95bfd8a182a09a99" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44693 } }
I20260927 10:59:01.861626 23167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:01.862656 23178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8177adde9826459e95bfd8a182a09a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8177adde9826459e95bfd8a182a09a99" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44693 } } }
I20260927 10:59:01.862782 23178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:01.863090 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8177adde9826459e95bfd8a182a09a99. Latest consensus state: current_term: 1 leader_uuid: "8177adde9826459e95bfd8a182a09a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8177adde9826459e95bfd8a182a09a99" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44693 } } }
I20260927 10:59:01.863160 23186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:59:01.863229 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:01.866187 23186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:59:01.871716 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:01.860500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39611 (local address 127.1.46.254:44693)
0927 10:59:01.860601 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:01.860605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:01.863823 (+ 3218us) server_negotiation.cc:407] Connection header received
0927 10:59:01.863935 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:01.863938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:01.863992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:01.864055 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:01.864908 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:01.865680 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:01.866736 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:01.867047 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:01.867117 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:01.868602 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:01.868607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:01.868610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:01.868638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:01.869941 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:01.870349 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:01.870355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:01.870359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:01.870436 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:01.870729 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:01.870735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:01.870737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:01.870932 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:01.871058 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:01.871439 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 10:59:01.871556 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:59:01.872640 23186 catalog_manager.cc:1347] Generated new cluster ID: 3507a18548bb4d0b8618d284b01653de
I20260927 10:59:01.872745 23186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:59:01.892597 23186 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:59:01.893486 23186 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:59:01.904995 23186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8177adde9826459e95bfd8a182a09a99: Generated new TSK 0
I20260927 10:59:01.905718 23186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:59:01.916986 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23099
2026-09-27T10:59:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:59:01 dist-test-slave-cffh krb5kdc[23198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:59:01 dist-test-slave-cffh krb5kdc[23198](info): set up 2 sockets
Sep 27 10:59:01 dist-test-slave-cffh krb5kdc[23198](info): commencing operation
krb5kdc: starting...
W20260927 10:59:02.958792 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:59:03 dist-test-slave-cffh krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:59:03.071892 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:35989
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:59:03.178018 23211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:03.178270 23211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:03.178314 23211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:03.182097 23211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:03.182156 23211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:03.182179 23211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:03.182199 23211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:03.182216 23211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:03.187948 23211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:03.188911 23211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:03.189859 23211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:59:03.198341 23216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:03.198428 23211 server_base.cc:1034] running on GCE node
W20260927 10:59:03.198475 23217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:03.198356 23219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:03.199169 23211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:03.200227 23211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:03.201423 23211 hybrid_clock.cc:648] HybridClock initialized: now 1790506743201387 us; error 47 us; skew 500 ppm
Sep 27 10:59:03 dist-test-slave-cffh krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506743, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:03.204097 23211 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:59:03.204932 23211 webserver.cc:460] Webserver started at http://127.1.46.254:40563/ using document root <none> and password file <none>
I20260927 10:59:03.205452 23211 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:03.205542 23211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:03.205794 23211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:03.207914 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "775233b8557649c684e932b8459df2c5"
format_stamp: "Formatted at 2026-09-27 10:59:03 on dist-test-slave-cffh"
server_key: "bf6347fa6b58867f97aaa393b996203d"
server_key_iv: "603e093f673ed799e582c4d5c63bd7e4"
server_key_version: "encryptionkey@0"
I20260927 10:59:03.208715 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "775233b8557649c684e932b8459df2c5"
format_stamp: "Formatted at 2026-09-27 10:59:03 on dist-test-slave-cffh"
server_key: "bf6347fa6b58867f97aaa393b996203d"
server_key_iv: "603e093f673ed799e582c4d5c63bd7e4"
server_key_version: "encryptionkey@0"
I20260927 10:59:03.212427 23211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:59:03.214794 23226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:03.216065 23211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:59:03.216238 23211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "775233b8557649c684e932b8459df2c5"
format_stamp: "Formatted at 2026-09-27 10:59:03 on dist-test-slave-cffh"
server_key: "bf6347fa6b58867f97aaa393b996203d"
server_key_iv: "603e093f673ed799e582c4d5c63bd7e4"
server_key_version: "encryptionkey@0"
I20260927 10:59:03.216369 23211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:03.231976 23211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:03.235294 23211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:03.235483 23211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:03.242153 23211 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:35989
I20260927 10:59:03.242182 23278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:35989 every 8 connection(s)
I20260927 10:59:03.243139 23211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:59:03.245677 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:03.247063 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23211
I20260927 10:59:03.247200 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:59:03.247509 1211 external_mini_cluster.cc:1428] Setting key 95496dd04172ac55bd8089b993bc0a17
I20260927 10:59:03.251929 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5: Bootstrap starting.
I20260927 10:59:03.254062 23279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5: Neither blocks nor log segments found. Creating new log.
I20260927 10:59:03.254973 23279 log.cc:824] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:03.256494 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:03.248861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41773 (local address 127.1.46.254:35989)
0927 10:59:03.249271 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:03.249286 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:03.249322 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:59:03.250314 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:03.250332 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:03.250726 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:03.251050 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:03.251422 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:03.252542 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:03.253416 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:03.253868 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:03.254034 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:03.254872 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:03.254877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:03.254889 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:59:03.255291 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:03.255377 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:59:03.255410 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:03.255512 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:03.255666 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 10:59:03.255854 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260927 10:59:03.257277 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5: No bootstrap required, opened a new log
I20260927 10:59:03.259290 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:59:03.261487 23279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775233b8557649c684e932b8459df2c5" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35989 } }
I20260927 10:59:03.261830 23279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:59:03.261921 23279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 775233b8557649c684e932b8459df2c5, State: Initialized, Role: FOLLOWER
I20260927 10:59:03.262413 23279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775233b8557649c684e932b8459df2c5" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35989 } }
I20260927 10:59:03.262574 23279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:59:03.262687 23279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:59:03.262794 23279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:03.263685 23279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775233b8557649c684e932b8459df2c5" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35989 } }
I20260927 10:59:03.264052 23279 leader_election.cc:302] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 775233b8557649c684e932b8459df2c5; no voters:
I20260927 10:59:03.264402 23279 leader_election.cc:288] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:59:03.264487 23289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:59:03.264978 23289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [term 1 LEADER]: Becoming Leader. State: Replica: 775233b8557649c684e932b8459df2c5, State: Running, Role: LEADER
I20260927 10:59:03.265440 23289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775233b8557649c684e932b8459df2c5" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35989 } }
I20260927 10:59:03.265811 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:03.261625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43691 (local address 127.1.46.254:35989)
0927 10:59:03.261748 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:03.261755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:03.261889 (+ 134us) server_negotiation.cc:407] Connection header received
0927 10:59:03.262011 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:03.262015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:03.262060 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:03.262143 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:03.262401 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:03.263098 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:03.263949 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:03.264245 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:03.264366 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:03.265170 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:03.265174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:03.265176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:59:03.265186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:03.265209 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:59:03.265219 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:03.265279 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:03.265616 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 10:59:03.265667 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:59:03.265870 23279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:03.267513 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 775233b8557649c684e932b8459df2c5. Latest consensus state: current_term: 1 leader_uuid: "775233b8557649c684e932b8459df2c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775233b8557649c684e932b8459df2c5" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35989 } } }
I20260927 10:59:03.267798 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "775233b8557649c684e932b8459df2c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775233b8557649c684e932b8459df2c5" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 35989 } } }
I20260927 10:59:03.267927 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:03.268026 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5 [sys.catalog]: This master's current role is: LEADER
W20260927 10:59:03.270511 23303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:59:03.270624 23303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:59:03.270704 23304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:59:03.271440 23304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:59:03.276619 23304 catalog_manager.cc:1347] Generated new cluster ID: be833cd93b2c44ad9da790c7387dab75
I20260927 10:59:03.276677 23304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:59:03.284366 23304 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:59:03.285194 23304 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:59:03.290803 23304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 775233b8557649c684e932b8459df2c5: Generated new TSK 0
I20260927 10:59:03.291381 23304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:59:03.298403 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23211
2026-09-27T10:59:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:59:03 dist-test-slave-cffh krb5kdc[23310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:59:03 dist-test-slave-cffh krb5kdc[23310](info): set up 2 sockets
Sep 27 10:59:03 dist-test-slave-cffh krb5kdc[23310](info): commencing operation
krb5kdc: starting...
W20260927 10:59:04.342747 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:59:04 dist-test-slave-cffh krb5kdc[23310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:59:04.459553 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:33781
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:45819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:33781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:59:04.565606 23323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:04.565838 23323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:04.565876 23323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:04.569834 23323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:04.569886 23323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:04.569904 23323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:04.569919 23323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:04.569934 23323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:04.574869 23323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:45819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:33781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:33781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:04.575685 23323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:04.576573 23323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:59:04.584982 23329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:04.585009 23328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:04.584983 23331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:04.585219 23323 server_base.cc:1034] running on GCE node
I20260927 10:59:04.585840 23323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:04.586915 23323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:04.588135 23323 hybrid_clock.cc:648] HybridClock initialized: now 1790506744588117 us; error 27 us; skew 500 ppm
Sep 27 10:59:04 dist-test-slave-cffh krb5kdc[23310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506744, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:04.590946 23323 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:59:04.591802 23323 webserver.cc:460] Webserver started at http://127.1.46.254:44709/ using document root <none> and password file <none>
I20260927 10:59:04.592305 23323 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:04.592391 23323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:04.592604 23323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:04.594456 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "8c2c8021baaa472b884ec6800e16fac4"
format_stamp: "Formatted at 2026-09-27 10:59:04 on dist-test-slave-cffh"
server_key: "c965f3fe88d9d57431421f0d55a5c411"
server_key_iv: "634e881bc2b675c9d0b7a68cd8892b46"
server_key_version: "encryptionkey@0"
I20260927 10:59:04.595045 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "8c2c8021baaa472b884ec6800e16fac4"
format_stamp: "Formatted at 2026-09-27 10:59:04 on dist-test-slave-cffh"
server_key: "c965f3fe88d9d57431421f0d55a5c411"
server_key_iv: "634e881bc2b675c9d0b7a68cd8892b46"
server_key_version: "encryptionkey@0"
I20260927 10:59:04.598446 23323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:59:04.600682 23338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:04.601812 23323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:59:04.601948 23323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "8c2c8021baaa472b884ec6800e16fac4"
format_stamp: "Formatted at 2026-09-27 10:59:04 on dist-test-slave-cffh"
server_key: "c965f3fe88d9d57431421f0d55a5c411"
server_key_iv: "634e881bc2b675c9d0b7a68cd8892b46"
server_key_version: "encryptionkey@0"
I20260927 10:59:04.602073 23323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:04.606374 23323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:04.609479 23323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:04.609666 23323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:04.616565 23323 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:33781
I20260927 10:59:04.616590 23390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:33781 every 8 connection(s)
I20260927 10:59:04.617393 23323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:59:04.619879 23391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:04.624980 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23323
I20260927 10:59:04.625133 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:59:04.625212 23391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4: Bootstrap starting.
I20260927 10:59:04.625465 1211 external_mini_cluster.cc:1428] Setting key e34fd9d4a2f3ff5e1b6835277f8fee3b
I20260927 10:59:04.627641 23391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4: Neither blocks nor log segments found. Creating new log.
I20260927 10:59:04.628338 23391 log.cc:824] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:04.630483 23391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4: No bootstrap required, opened a new log
I20260927 10:59:04.630692 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:04.626769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35793 (local address 127.1.46.254:33781)
0927 10:59:04.627171 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:04.627184 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:04.627217 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:59:04.627898 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:04.627916 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:04.628187 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:04.628526 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:04.629360 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:04.629366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:04.629381 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:59:04.629392 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:04.629450 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:59:04.629474 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:04.629583 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:04.629690 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 10:59:04.629843 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260927 10:59:04.633261 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:59:04.634495 23391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2c8021baaa472b884ec6800e16fac4" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 33781 } }
I20260927 10:59:04.634812 23391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:59:04.634874 23391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2c8021baaa472b884ec6800e16fac4, State: Initialized, Role: FOLLOWER
I20260927 10:59:04.635315 23391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2c8021baaa472b884ec6800e16fac4" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 33781 } }
I20260927 10:59:04.635515 23391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:59:04.635617 23391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:59:04.635747 23391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:04.637024 23391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2c8021baaa472b884ec6800e16fac4" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 33781 } }
I20260927 10:59:04.637246 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:04.635519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34449 (local address 127.1.46.254:33781)
0927 10:59:04.635621 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:04.635627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:04.635908 (+ 281us) server_negotiation.cc:407] Connection header received
0927 10:59:04.636039 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:04.636041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:04.636069 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:04.636130 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:04.636832 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:04.636834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:04.636836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:59:04.636846 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:04.636870 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:59:04.636880 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:04.636944 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:04.637072 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 10:59:04.637101 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 10:59:04.637403 23391 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c2c8021baaa472b884ec6800e16fac4; no voters:
I20260927 10:59:04.637729 23391 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:59:04.637820 23401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:59:04.638003 23401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [term 1 LEADER]: Becoming Leader. State: Replica: 8c2c8021baaa472b884ec6800e16fac4, State: Running, Role: LEADER
I20260927 10:59:04.638291 23401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2c8021baaa472b884ec6800e16fac4" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 33781 } }
I20260927 10:59:04.638990 23391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:04.639940 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c2c8021baaa472b884ec6800e16fac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2c8021baaa472b884ec6800e16fac4" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 33781 } } }
I20260927 10:59:04.639976 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2c8021baaa472b884ec6800e16fac4. Latest consensus state: current_term: 1 leader_uuid: "8c2c8021baaa472b884ec6800e16fac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2c8021baaa472b884ec6800e16fac4" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 33781 } } }
I20260927 10:59:04.640120 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:04.640120 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:04.640501 23410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:59:04.643326 23410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:59:04.648610 23410 catalog_manager.cc:1347] Generated new cluster ID: edc4c58027ce4c3690c5aee2dc185e38
I20260927 10:59:04.648671 23410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:59:04.657269 23410 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:59:04.657544 23410 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:59:04.675594 23410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c2c8021baaa472b884ec6800e16fac4: Generated new TSK 0
I20260927 10:59:04.676230 23410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:59:04.683468 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23323
2026-09-27T10:59:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1384 ms)
[----------] 18 tests from AuthTokenIssuingTest (24735 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:59:04 dist-test-slave-cffh krb5kdc[23422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:59:04 dist-test-slave-cffh krb5kdc[23422](info): set up 2 sockets
Sep 27 10:59:04 dist-test-slave-cffh krb5kdc[23422](info): commencing operation
krb5kdc: starting...
W20260927 10:59:05.707054 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:59:05 dist-test-slave-cffh krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:59:05.819464 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44829
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:44829,127.1.46.253:40227,127.1.46.252:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:59:05.923144 23435 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:59:05.923442 23435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:05.923492 23435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:05.923530 23435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:05.927608 23435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:05.927656 23435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:05.927675 23435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:05.927691 23435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:05.927706 23435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:05.932648 23435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:44829,127.1.46.253:40227,127.1.46.252:40333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:05.933545 23435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:05.934497 23435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:59:05.942688 23443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:05.942720 23435 server_base.cc:1034] running on GCE node
W20260927 10:59:05.942694 23441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:05.942688 23440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:05.943445 23435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:05.944394 23435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:05.945547 23435 hybrid_clock.cc:648] HybridClock initialized: now 1790506745945531 us; error 28 us; skew 500 ppm
Sep 27 10:59:05 dist-test-slave-cffh krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:05.948195 23435 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:59:05.948968 23435 webserver.cc:460] Webserver started at http://127.1.46.254:41795/ using document root <none> and password file <none>
I20260927 10:59:05.949450 23435 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:05.949503 23435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:05.949653 23435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:05.951596 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "bfcd2e1b8ede42089886e5c4fde97b40"
format_stamp: "Formatted at 2026-09-27 10:59:05 on dist-test-slave-cffh"
server_key: "3bcf41b5927de93ed6a50b1794c6d1ef"
server_key_iv: "97faa461e9c54db6b41dbba2830720c3"
server_key_version: "encryptionkey@0"
I20260927 10:59:05.952076 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "bfcd2e1b8ede42089886e5c4fde97b40"
format_stamp: "Formatted at 2026-09-27 10:59:05 on dist-test-slave-cffh"
server_key: "3bcf41b5927de93ed6a50b1794c6d1ef"
server_key_iv: "97faa461e9c54db6b41dbba2830720c3"
server_key_version: "encryptionkey@0"
I20260927 10:59:05.955497 23435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:59:05.957546 23450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:05.958803 23435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:59:05.958945 23435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "bfcd2e1b8ede42089886e5c4fde97b40"
format_stamp: "Formatted at 2026-09-27 10:59:05 on dist-test-slave-cffh"
server_key: "3bcf41b5927de93ed6a50b1794c6d1ef"
server_key_iv: "97faa461e9c54db6b41dbba2830720c3"
server_key_version: "encryptionkey@0"
I20260927 10:59:05.959038 23435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:05.977847 23435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:05.980520 23435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:05.980728 23435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:05.987562 23435 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:44829
I20260927 10:59:05.987846 23502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:44829 every 8 connection(s)
I20260927 10:59:05.988438 23435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:59:05.991145 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:05.994550 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23435
I20260927 10:59:05.994707 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:59:05.994669 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:05.995021 1211 external_mini_cluster.cc:1428] Setting key 11e56b9fb857c314fc8f213dbeecfbc5
WARNING: no policy specified for kudu/127.1.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.253@KRBTEST.COM" created.
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:59:06.010807 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:05.997104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44829 (local address 127.0.0.1:42628)
0927 10:59:05.997614 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:05.997651 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:05.998481 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:05.999122 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:05.999127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:05.999516 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:05.999729 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:05.999740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.000851 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.000854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.001515 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.001522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.002115 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.002118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.002229 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.002816 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.002837 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.006072 (+ 3235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.008081 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.008091 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.008099 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.008291 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.008630 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.008635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.008638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.008740 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.009278 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.009283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.009408 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.009561 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.009890 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":115,"thread_start_us":170,"threads_started":1}
I20260927 10:59:06.010807 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:05.998050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.1.46.254:44829)
0927 10:59:05.998363 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:05.998378 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:05.998417 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:59:05.998785 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:05.998789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:05.998955 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:05.999124 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:05.999886 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.000714 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.001626 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.001970 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.002133 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.006219 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.006224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.006232 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.006251 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.007988 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.008402 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.008407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.008410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.008511 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.008845 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.008851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.008855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.009166 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.009290 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.009616 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.009836 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
I20260927 10:59:06.013564 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:06.014293 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.013966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51356)
0927 10:59:06.014108 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.014206 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 10:59:06.014526 23454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
I20260927 10:59:06.015400 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.015144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51358)
0927 10:59:06.015274 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.015333 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:59:06.015553 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:40227: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab.
Entry for principal kudu/127.1.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab.
I20260927 10:59:06.019677 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:40227
--webserver_interface=127.1.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab
--principal=kudu/127.1.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:44829,127.1.46.253:40227,127.1.46.252:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:59:06.064791 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } attempt: 1
I20260927 10:59:06.065629 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.065318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51362)
0927 10:59:06.065491 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.065555 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 10:59:06.066337 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.066158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51364)
0927 10:59:06.066247 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.066288 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:59:06.066495 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:40227: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
W20260927 10:59:06.124420 23510 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:59:06.124804 23510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:06.124869 23510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:06.124950 23510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:06.128969 23510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:06.129020 23510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:06.129040 23510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:06.129055 23510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:06.129070 23510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:06.133996 23510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:44829,127.1.46.253:40227,127.1.46.252:40333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab
--principal=kudu/127.1.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:06.134629 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } attempt: 2
I20260927 10:59:06.134981 23510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:06.135442 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.135166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51370)
0927 10:59:06.135307 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.135370 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:59:06.136158 23510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:59:06.136173 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.135978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51374)
0927 10:59:06.136085 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.136125 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:59:06.136353 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:40227: Network error: Client connection negotiation failed: client connection to 127.1.46.253:40227: connect: Connection refused (error 111)
W20260927 10:59:06.144162 23515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:06.144166 23518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:06.144236 23516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:06.144479 23510 server_base.cc:1034] running on GCE node
I20260927 10:59:06.144976 23510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:06.146039 23510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:06.147208 23510 hybrid_clock.cc:648] HybridClock initialized: now 1790506746147169 us; error 52 us; skew 500 ppm
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:06.150053 23510 init.cc:377] Logged in from keytab as kudu/127.1.46.253@KRBTEST.COM (short username kudu)
I20260927 10:59:06.151032 23510 webserver.cc:460] Webserver started at http://127.1.46.253:39073/ using document root <none> and password file <none>
I20260927 10:59:06.151593 23510 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:06.151660 23510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:06.151826 23510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:06.153791 23510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data/instance:
uuid: "567ce9cf9e974862ae450acd91943ed6"
format_stamp: "Formatted at 2026-09-27 10:59:06 on dist-test-slave-cffh"
server_key: "527bca8d2a3133298cf29dd7f90f340f"
server_key_iv: "61c3dc7ae96ec65107723d2af8160756"
server_key_version: "encryptionkey@0"
I20260927 10:59:06.154376 23510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/wal/instance:
uuid: "567ce9cf9e974862ae450acd91943ed6"
format_stamp: "Formatted at 2026-09-27 10:59:06 on dist-test-slave-cffh"
server_key: "527bca8d2a3133298cf29dd7f90f340f"
server_key_iv: "61c3dc7ae96ec65107723d2af8160756"
server_key_version: "encryptionkey@0"
I20260927 10:59:06.157927 23510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:59:06.160267 23525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:06.161742 23510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:59:06.161926 23510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/wal
uuid: "567ce9cf9e974862ae450acd91943ed6"
format_stamp: "Formatted at 2026-09-27 10:59:06 on dist-test-slave-cffh"
server_key: "527bca8d2a3133298cf29dd7f90f340f"
server_key_iv: "61c3dc7ae96ec65107723d2af8160756"
server_key_version: "encryptionkey@0"
I20260927 10:59:06.162030 23510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:06.178521 23510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:06.181648 23510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:06.181882 23510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:06.189476 23510 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.253:40227
I20260927 10:59:06.189759 23577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.253:40227 every 8 connection(s)
I20260927 10:59:06.190543 23510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
I20260927 10:59:06.193233 23578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:06.194960 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23510
I20260927 10:59:06.195099 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-1/wal/instance
I20260927 10:59:06.195384 1211 external_mini_cluster.cc:1428] Setting key 7851e0a7001b1903a6d8b7fdd3251e25
I20260927 10:59:06.197232 23578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.252@KRBTEST.COM" created.
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:59:06.212541 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.199137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.1.46.254:44829)
0927 10:59:06.199276 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.199281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.200329 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 10:59:06.201303 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.201307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.201367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.201484 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.202850 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.203458 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.204847 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.205148 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.205285 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.208540 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.208543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.208545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.208568 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.210153 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.210707 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.210713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.210718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.210815 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.211177 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.211184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.211187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.211437 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.211563 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.212196 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.212343 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:59:06.213244 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.199619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44829 (local address 127.0.0.1:42644)
0927 10:59:06.200153 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.200197 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.201074 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.201650 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.201657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.202134 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.202675 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.202691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.203596 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.203601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.204714 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.204724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.205310 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.205314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.205585 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.206219 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.206247 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.208356 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.210285 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.210289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.210300 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.210553 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.210915 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.210917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.210919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.211063 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.211653 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.211659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.211918 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.212079 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.212315 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":145,"threads_started":1}
I20260927 10:59:06.214661 23578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab.
Entry for principal kudu/127.1.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab.
I20260927 10:59:06.221531 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:40333
--webserver_interface=127.1.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab
--principal=kudu/127.1.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:44829,127.1.46.253:40227,127.1.46.252:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:59:06.224591 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.215316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.1.46.253:40227)
0927 10:59:06.215612 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.215621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.215639 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:59:06.215672 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.215675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.215847 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.215953 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.216390 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.217118 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.218105 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.218448 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.218537 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.220858 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.220870 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.220891 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.220913 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.222771 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.223280 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.223285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.223286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.223364 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.223686 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.223688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.223689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.223905 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.224006 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.224323 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.224437 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":123,"threads_started":1}
I20260927 10:59:06.224684 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.215188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51380)
0927 10:59:06.215388 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.215406 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.215486 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.215983 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.215986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.216189 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.216274 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.216279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.217260 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.217274 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.217998 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.218007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.218618 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.218624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.218755 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.219196 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.219216 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.220708 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.222871 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.222876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.222879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.223147 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.223473 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.223478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.223481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.223574 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.224006 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.224012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.224118 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.224234 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.224378 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 10:59:06.226745 23578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:06.227542 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.227224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48780)
0927 10:59:06.227379 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.227465 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:59:06.227514 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } attempt: 3
W20260927 10:59:06.227725 23527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
I20260927 10:59:06.228806 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.228610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48792)
0927 10:59:06.228699 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.228747 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 10:59:06.229040 23578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:40333: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:59:06.236028 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.227931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51392)
0927 10:59:06.228046 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.228062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.228140 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.228565 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.228567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.228743 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.228894 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.228904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.229667 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.229672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.230208 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.230216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.230754 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.230757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.230900 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.231344 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.231354 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.232584 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.234599 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.234605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.234610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.234819 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.235095 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.235099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.235101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.235185 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.235535 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.235538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.235638 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.235742 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.235871 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:59:06.236039 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.228222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.1.46.253:40227)
0927 10:59:06.228346 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.228350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.228363 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:59:06.228393 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.228396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.228436 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.228539 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.228991 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.229546 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.230346 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.230642 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.230752 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.232710 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.232714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.232717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.232748 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.234465 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.234914 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.234920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.234922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.234997 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.235270 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.235275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.235280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.235439 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.235542 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.235800 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.235909 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:59:06.237068 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:06.237712 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.237507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48800)
0927 10:59:06.237615 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.237659 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:59:06.238417 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.238180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48812)
0927 10:59:06.238324 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.238372 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 10:59:06.238560 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:40333: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
I20260927 10:59:06.269773 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } attempt: 1
I20260927 10:59:06.270707 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.270309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48822)
0927 10:59:06.270491 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.270559 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 10:59:06.271476 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.271232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48830)
0927 10:59:06.271356 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.271416 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:59:06.271620 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:40333: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
I20260927 10:59:06.278318 23578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } attempt: 1
I20260927 10:59:06.279052 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.278751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48834)
0927 10:59:06.278885 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.278948 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 10:59:06.279759 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.279568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48842)
0927 10:59:06.279665 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.279708 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:59:06.279950 23578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:40333: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
W20260927 10:59:06.333763 23584 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:59:06.334007 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:06.334030 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:06.334131 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:06.338189 23584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:06.338238 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:06.338256 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:06.338274 23584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:06.338289 23584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:06.343465 23584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:44829,127.1.46.253:40227,127.1.46.252:40333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab
--principal=kudu/127.1.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:40333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:06.344389 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:06.345306 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:59:06.346792 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } attempt: 2
I20260927 10:59:06.347718 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.347436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48854)
0927 10:59:06.347573 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.347642 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 10:59:06.348111 23578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } attempt: 2
I20260927 10:59:06.348647 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.348341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48868)
0927 10:59:06.348518 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.348576 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:59:06.348907 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.348592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48876)
0927 10:59:06.348744 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.348802 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 10:59:06.349090 23503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:40333: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
I20260927 10:59:06.349738 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.349464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48882)
0927 10:59:06.349580 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.349645 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 10:59:06.349877 23578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:40333: Network error: Client connection negotiation failed: client connection to 127.1.46.252:40333: connect: Connection refused (error 111)
W20260927 10:59:06.353829 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:06.353838 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:06.354007 23584 server_base.cc:1034] running on GCE node
W20260927 10:59:06.357184 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:06.357909 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:06.359282 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:06.360453 23584 hybrid_clock.cc:648] HybridClock initialized: now 1790506746360423 us; error 42 us; skew 500 ppm
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:06.363920 23584 init.cc:377] Logged in from keytab as kudu/127.1.46.252@KRBTEST.COM (short username kudu)
I20260927 10:59:06.364795 23584 webserver.cc:460] Webserver started at http://127.1.46.252:32809/ using document root <none> and password file <none>
I20260927 10:59:06.365378 23584 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:06.365437 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:06.365648 23584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:06.367812 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data/instance:
uuid: "2409d437379f4ee3be22a7b1bfb0ee2c"
format_stamp: "Formatted at 2026-09-27 10:59:06 on dist-test-slave-cffh"
server_key: "d8a0d313d45eab312827c4bb6cefbeb6"
server_key_iv: "ca0eb73607534d23d308e8682462532a"
server_key_version: "encryptionkey@0"
I20260927 10:59:06.368386 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/wal/instance:
uuid: "2409d437379f4ee3be22a7b1bfb0ee2c"
format_stamp: "Formatted at 2026-09-27 10:59:06 on dist-test-slave-cffh"
server_key: "d8a0d313d45eab312827c4bb6cefbeb6"
server_key_iv: "ca0eb73607534d23d308e8682462532a"
server_key_version: "encryptionkey@0"
I20260927 10:59:06.371963 23584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:59:06.374300 23600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:06.375589 23584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:59:06.375734 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/wal
uuid: "2409d437379f4ee3be22a7b1bfb0ee2c"
format_stamp: "Formatted at 2026-09-27 10:59:06 on dist-test-slave-cffh"
server_key: "d8a0d313d45eab312827c4bb6cefbeb6"
server_key_iv: "ca0eb73607534d23d308e8682462532a"
server_key_version: "encryptionkey@0"
I20260927 10:59:06.375837 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:06.397022 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:06.399982 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:06.400169 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:06.406890 23584 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.252:40333
I20260927 10:59:06.407150 23652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.252:40333 every 8 connection(s)
I20260927 10:59:06.407806 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
I20260927 10:59:06.408008 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23584
I20260927 10:59:06.408166 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790506472751632-1211-0/minicluster-data/master-2/wal/instance
I20260927 10:59:06.408540 1211 external_mini_cluster.cc:1428] Setting key f28af939fe74811b020dee9146c5949c
I20260927 10:59:06.410679 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:06.414147 23653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:59:06.423156 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.409835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.1.46.252:40333)
0927 10:59:06.410226 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.410241 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.410291 (+ 50us) server_negotiation.cc:407] Connection header received
0927 10:59:06.411055 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.411073 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.411386 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.411703 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.412571 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.413567 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.414478 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.415010 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.415186 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.418163 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.418167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.418175 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.418198 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.420011 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.420458 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.420462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.420466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.420577 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.420882 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.420886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.420887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.421320 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.421460 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.421808 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.422058 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
I20260927 10:59:06.426263 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.415806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.1.46.254:44829)
0927 10:59:06.415948 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.415952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.416208 (+ 256us) server_negotiation.cc:407] Connection header received
0927 10:59:06.416354 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.416356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.416396 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.416462 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.417232 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.417867 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.418832 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.419167 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.419311 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.421718 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.421722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.421724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.421747 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.424426 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.424925 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.424930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.424933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.425018 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.425336 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.425341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.425344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.425547 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.425650 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.426080 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.426163 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:59:06.426347 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.415703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:44829 (local address 127.0.0.1:42652)
0927 10:59:06.416063 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.416113 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.416239 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.416548 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.416556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.416991 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.417120 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.417126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.417985 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.417988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.418708 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.418718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.419335 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.419339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.419429 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.419935 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.419946 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.421575 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.424532 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.424541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.424546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.424831 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.425131 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.425134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.425136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.425228 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.425651 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.425656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.425797 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.425982 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.426160 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
I20260927 10:59:06.427402 23653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:59:06.434689 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.425185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.1.46.254:44829)
0927 10:59:06.425504 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.425514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.425525 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:59:06.425555 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.425558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.425613 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.425714 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.426425 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.427224 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.428215 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.428555 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.428651 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.431203 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.431206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.431208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.431227 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.432730 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.433424 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.433430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.433434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.433535 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.433790 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.433795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.433798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.433966 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.434158 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.434338 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.434450 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":88,"threads_started":1}
I20260927 10:59:06.436143 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.427825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:40227 (local address 127.0.0.1:51400)
0927 10:59:06.427948 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.427962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.428029 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.428344 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.428346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.428465 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.428562 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.428567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.429281 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.429287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.430164 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.430175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.430784 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.430788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.430886 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.431307 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.431324 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.432716 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.434523 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.434527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.434530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.434764 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.435099 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.435105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.435107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.435200 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.435647 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.435653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.435754 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.435862 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.435979 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 10:59:06.436601 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.427972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.1.46.253:40227)
0927 10:59:06.428102 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.428107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.428120 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:59:06.428184 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.428189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.428235 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.428311 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.428697 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.429182 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.430281 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.430659 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.430765 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.432859 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.432864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.432867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.432892 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.434398 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.434898 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.434903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.434905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.434989 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.435308 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.435310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.435311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.435542 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.435640 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.436329 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.436447 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:59:06.437274 23653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:59:06.441031 23578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } attempt: 3
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:59:06.445076 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.435906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.1.46.253:40227)
0927 10:59:06.436236 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.436247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.436262 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:59:06.436291 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.436294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.436329 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.436415 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.437194 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.437934 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.438670 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.438928 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.439025 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.441345 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.441349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.441353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.441378 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.443371 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.443783 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.443786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.443788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.443870 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.444159 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.444168 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.444170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.444439 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.444616 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.444787 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.444907 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":104,"threads_started":1}
I20260927 10:59:06.445158 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.437744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.1.46.252:40333)
0927 10:59:06.437837 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.437841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.437854 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:59:06.437945 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.437947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.437975 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.438060 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.438364 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.438910 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.439741 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.439950 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.440050 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.442005 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.442009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.442013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.442041 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.443624 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.443997 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.444000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.444002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.444070 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.444355 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.444358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.444360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.444552 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.444634 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.444931 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.445021 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:59:06.445467 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.437712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48912)
0927 10:59:06.437800 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.437812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.437872 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.438067 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.438070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.438210 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.438261 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.438265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.439000 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.439006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.439622 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.439631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.440048 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.440051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.440133 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.440523 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.440538 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.441851 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.443737 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.443742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.443744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.443907 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.444152 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.444154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.444156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.444240 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.444668 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.444672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.444752 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.445255 (+ 503us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.445364 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":3}
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:59:06.448637 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: Bootstrap starting.
I20260927 10:59:06.448702 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:59:06.450608 23503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } attempt: 3
I20260927 10:59:06.451002 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.441364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48922)
0927 10:59:06.441474 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.441489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.441559 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.442735 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.442737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.442907 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.443019 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.443026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.443997 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.444000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.444866 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.444876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.445574 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.445577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.445653 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.446097 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.446112 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.447484 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.449483 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.449486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.449488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.449731 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.450057 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.450060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.450062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.450149 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.450536 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.450539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.450646 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.450747 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.450875 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":24}
I20260927 10:59:06.451273 23653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: Neither blocks nor log segments found. Creating new log.
I20260927 10:59:06.450042 23477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bfcd2e1b8ede42089886e5c4fde97b40"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42654
I20260927 10:59:06.451447 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.442226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.1.46.252:40333)
0927 10:59:06.442501 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.442507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.442517 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:59:06.442541 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.442543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.442625 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.442722 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.443154 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.443893 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.445011 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.445452 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.445869 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.447596 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.447600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.447603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.447625 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.449373 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.449852 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.449855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.449857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.449961 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.450247 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.450251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.450252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.450435 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.450529 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.451176 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.451283 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":217,"thread_start_us":124,"threads_started":1}
I20260927 10:59:06.452153 23653 log.cc:824] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:06.453913 23578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: Bootstrap starting.
I20260927 10:59:06.453956 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: No bootstrap required, opened a new log
I20260927 10:59:06.456146 23578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: Neither blocks nor log segments found. Creating new log.
Sep 27 10:59:06 dist-test-slave-cffh krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506745, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:59:06.456846 23578 log.cc:824] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:06.458287 23653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
I20260927 10:59:06.458542 23653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2409d437379f4ee3be22a7b1bfb0ee2c, State: Initialized, Role: FOLLOWER
I20260927 10:59:06.458966 23578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: No bootstrap required, opened a new log
I20260927 10:59:06.458899 23653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
I20260927 10:59:06.459726 23663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.459918 23663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:06.460029 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.451699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:40333 (local address 127.0.0.1:48936)
0927 10:59:06.451789 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:06.451802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:06.451876 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:06.452176 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:06.452179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:06.452335 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:06.452440 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.452446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.453397 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.453400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.454075 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:06.454084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.454606 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.454609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:06.454680 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.455154 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:06.455167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:06.456607 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:06.458504 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.458509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.458512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.458771 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.459114 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:06.459117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:06.459119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:06.459212 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:06.459600 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:06.459603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:06.459685 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 10:59:06.459790 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:59:06.459902 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 10:59:06.460085 23653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:06.460443 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.451876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.1.46.252:40333)
0927 10:59:06.451952 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.451955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.451966 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:59:06.452015 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.452019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.452058 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.452134 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.452573 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.453293 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.454187 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.454484 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.454596 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.456741 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.456746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.456750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.456773 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.458398 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.458885 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.458888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.458890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.458996 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.459328 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.459330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.459332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.459495 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.459576 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.460199 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.460288 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 10:59:06.463235 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40: Bootstrap starting.
I20260927 10:59:06.463599 23578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
I20260927 10:59:06.465458 23503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40: Neither blocks nor log segments found. Creating new log.
W20260927 10:59:06.465672 23674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:59:06.465767 23674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:59:06.466221 23503 log.cc:824] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:06.463799 23584 master_runner.cc:186] Error getting master registration for 127.1.46.254:44829: OK, instance_id { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" instance_seqno: 1790506745985678 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:59:06.467085 23584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:59:06.467177 23578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567ce9cf9e974862ae450acd91943ed6, State: Initialized, Role: FOLLOWER
I20260927 10:59:06.467485 23578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
I20260927 10:59:06.467904 23677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.468066 23677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:06.468582 23578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:06.468676 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40: No bootstrap required, opened a new log
W20260927 10:59:06.473106 23688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 10:59:06.472944 23503 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
W20260927 10:59:06.473243 23688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:59:06.473213 23503 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcd2e1b8ede42089886e5c4fde97b40, State: Initialized, Role: FOLLOWER
I20260927 10:59:06.473711 23503 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
I20260927 10:59:06.471660 23510 master_runner.cc:186] Error getting master registration for 127.1.46.254:44829: OK, instance_id { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" instance_seqno: 1790506745985678 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:59:06.475430 23510 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:59:06.475683 23503 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:06.475792 23477 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:59:06.475909 23477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:06.477535 23477 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
I20260927 10:59:06.478140 23690 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.478308 23690 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:06.478380 23477 leader_election.cc:288] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [CANDIDATE]: Term 1 election: Requested vote from peers 567ce9cf9e974862ae450acd91943ed6 (127.1.46.253:40227), 2409d437379f4ee3be22a7b1bfb0ee2c (127.1.46.252:40333)
I20260927 10:59:06.479423 23552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "567ce9cf9e974862ae450acd91943ed6"
I20260927 10:59:06.479635 23552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:06.481252 23552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcd2e1b8ede42089886e5c4fde97b40 in term 1.
I20260927 10:59:06.481647 23627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c"
I20260927 10:59:06.481863 23627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:59:06.482105 23456 tablet.cc:2369] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:59:06.482105 23701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:59:06.482682 23701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:59:06.483166 23627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcd2e1b8ede42089886e5c4fde97b40 in term 1.
I20260927 10:59:06.485726 23452 leader_election.cc:302] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2409d437379f4ee3be22a7b1bfb0ee2c, bfcd2e1b8ede42089886e5c4fde97b40; no voters:
I20260927 10:59:06.486002 23690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:59:06.486202 23690 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [term 1 LEADER]: Becoming Leader. State: Replica: bfcd2e1b8ede42089886e5c4fde97b40, State: Running, Role: LEADER
I20260927 10:59:06.486527 23690 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } }
I20260927 10:59:06.488161 23690 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfcd2e1b8ede42089886e5c4fde97b40. Latest consensus state: current_term: 1 leader_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.488263 23690 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:06.495183 23709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:59:06.496042 23709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:59:06.500620 23627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [term 1 FOLLOWER]: Refusing update from remote peer bfcd2e1b8ede42089886e5c4fde97b40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:59:06.500676 23552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [term 1 FOLLOWER]: Refusing update from remote peer bfcd2e1b8ede42089886e5c4fde97b40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:59:06.501299 23690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:59:06.505249 23677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfcd2e1b8ede42089886e5c4fde97b40. Latest consensus state: current_term: 1 leader_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.505360 23677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:06.505467 23707 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:59:06.508312 23663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfcd2e1b8ede42089886e5c4fde97b40. Latest consensus state: current_term: 1 leader_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.508422 23663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:06.511708 23707 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.511827 23707 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:06.512602 23677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.512703 23677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:06.513152 23713 mvcc.cc:205] Tried to move back new op lower bound from 7333915633660940288 to 7333915633612013568. Current Snapshot: MvccSnapshot[applied={T|T < 7333915633660940288}]
I20260927 10:59:06.514117 23714 mvcc.cc:205] Tried to move back new op lower bound from 7333915633660940288 to 7333915633612013568. Current Snapshot: MvccSnapshot[applied={T|T < 7333915633660940288}]
I20260927 10:59:06.518046 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.493911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.1.46.253:40227)
0927 10:59:06.494026 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.494031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.501580 (+ 7549us) server_negotiation.cc:407] Connection header received
0927 10:59:06.501748 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.501751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.501794 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.501841 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.503447 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.504291 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.508664 (+ 4373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.509036 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.509109 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.514338 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.514342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.514344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.514368 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.516250 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.516703 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.516706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.516708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.516828 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.517109 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.517112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.517114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.517315 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.517418 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.517775 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.517899 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:59:06.520258 23690 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.520367 23690 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:06.521257 23663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcd2e1b8ede42089886e5c4fde97b40" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 44829 } } peers { permanent_uuid: "567ce9cf9e974862ae450acd91943ed6" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 40227 } } peers { permanent_uuid: "2409d437379f4ee3be22a7b1bfb0ee2c" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 40333 } } }
I20260927 10:59:06.521400 23663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:06.521816 23716 mvcc.cc:205] Tried to move back new op lower bound from 7333915633660940288 to 7333915633612013568. Current Snapshot: MvccSnapshot[applied={T|T < 7333915633660940288}]
I20260927 10:59:06.523799 23709 catalog_manager.cc:1347] Generated new cluster ID: 7f28d420b49643c4975b96ddc0f4ab5d
I20260927 10:59:06.523854 23709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:59:06.523921 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.494681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.1.46.252:40333)
0927 10:59:06.494800 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.494804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.510019 (+ 15215us) server_negotiation.cc:407] Connection header received
0927 10:59:06.511422 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.511425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.511470 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.511518 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.512380 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.513128 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.514481 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.514842 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.514911 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.520185 (+ 5274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.520189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.520192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.520217 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.522005 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.522650 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.522654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.522656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.522739 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.523037 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.523041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.523043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.523233 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.523360 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.523672 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.523777 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 10:59:06.526666 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:06.495201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.1.46.254:44829)
0927 10:59:06.495318 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.495322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.518555 (+ 23233us) server_negotiation.cc:407] Connection header received
0927 10:59:06.518754 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.518757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.518813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.518865 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.519707 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.520477 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.521509 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:06.521866 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:06.521943 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:06.523481 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:06.523485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:06.523487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:06.523510 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:06.525140 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.525546 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.525549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.525551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.525616 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:06.525863 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:06.525866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:06.525867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:06.526033 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:06.526125 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:06.526430 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:59:06.526524 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:59:06.551192 23709 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:59:06.552091 23709 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:59:06.568432 23709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfcd2e1b8ede42089886e5c4fde97b40: Generated new TSK 0
I20260927 10:59:06.569167 23709 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:59:06.605314 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.603232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.1.46.253:40227)
0927 10:59:06.603371 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.603375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.604303 (+ 928us) server_negotiation.cc:407] Connection header received
0927 10:59:06.604494 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.604498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.604544 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.604596 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.605182 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51412: BlockingRecv error: recv got EOF from 127.0.0.1:51412 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 10:59:06.606673 23660 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.603789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.1.46.252:40333)
0927 10:59:06.603914 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.603919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.605769 (+ 1850us) server_negotiation.cc:407] Connection header received
0927 10:59:06.605962 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.605965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.606010 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.606058 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.606537 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48952: BlockingRecv error: recv got EOF from 127.0.0.1:48952 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 10:59:06.608464 23658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.607257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.1.46.254:44829)
0927 10:59:06.607364 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.607369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.607635 (+ 266us) server_negotiation.cc:407] Connection header received
0927 10:59:06.607790 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.607793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.607838 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.607886 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.608347 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42666: BlockingRecv error: recv got EOF from 127.0.0.1:42666 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 10:59:06.616503 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.614931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.1.46.253:40227)
0927 10:59:06.615029 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.615032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.615540 (+ 508us) server_negotiation.cc:407] Connection header received
0927 10:59:06.615786 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.615790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.615877 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.615945 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.616408 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51420: BlockingRecv error: recv got EOF from 127.0.0.1:51420 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 10:59:06.624706 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.623277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.1.46.253:40227)
0927 10:59:06.623372 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.623376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.623863 (+ 487us) server_negotiation.cc:407] Connection header received
0927 10:59:06.624051 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.624054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.624099 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.624150 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.624614 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51436: BlockingRecv error: recv got EOF from 127.0.0.1:51436 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 10:59:06.629944 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.628583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.1.46.253:40227)
0927 10:59:06.628674 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.628678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.629150 (+ 472us) server_negotiation.cc:407] Connection header received
0927 10:59:06.629319 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.629322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.629365 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.629411 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.629844 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51444: BlockingRecv error: recv got EOF from 127.0.0.1:51444 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 10:59:06.637092 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.635698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.1.46.253:40227)
0927 10:59:06.635814 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.635818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.636309 (+ 491us) server_negotiation.cc:407] Connection header received
0927 10:59:06.636471 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.636474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.636514 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.636575 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.637011 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51458: BlockingRecv error: recv got EOF from 127.0.0.1:51458 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 10:59:06.648312 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.646856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.1.46.253:40227)
0927 10:59:06.646959 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.646963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.647489 (+ 526us) server_negotiation.cc:407] Connection header received
0927 10:59:06.647661 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.647664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.647706 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.647754 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.648223 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51470: BlockingRecv error: recv got EOF from 127.0.0.1:51470 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 10:59:06.668394 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.666723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.1.46.253:40227)
0927 10:59:06.666858 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.666863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.667452 (+ 589us) server_negotiation.cc:407] Connection header received
0927 10:59:06.667656 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.667661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.667718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.667768 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.668290 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51478: BlockingRecv error: recv got EOF from 127.0.0.1:51478 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:59:06.704146 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.702790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.1.46.253:40227)
0927 10:59:06.702920 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.702925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.703197 (+ 272us) server_negotiation.cc:407] Connection header received
0927 10:59:06.703330 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.703333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.703385 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.703457 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.704020 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51492: BlockingRecv error: recv got EOF from 127.0.0.1:51492 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 10:59:06.771665 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.770536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.1.46.253:40227)
0927 10:59:06.770732 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.770737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.770915 (+ 178us) server_negotiation.cc:407] Connection header received
0927 10:59:06.771039 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.771042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.771093 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.771176 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.771571 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51506: BlockingRecv error: recv got EOF from 127.0.0.1:51506 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 10:59:06.903028 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:06.901837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.1.46.253:40227)
0927 10:59:06.902002 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:06.902005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:06.902061 (+ 56us) server_negotiation.cc:407] Connection header received
0927 10:59:06.902239 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:06.902244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:06.902312 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:06.902403 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:06.902911 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51522: BlockingRecv error: recv got EOF from 127.0.0.1:51522 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 10:59:07.162462 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:07.161322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.1.46.253:40227)
0927 10:59:07.161505 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:07.161511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:07.161527 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:59:07.161686 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:07.161690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:07.161749 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:07.161851 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:07.162312 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51526: BlockingRecv error: recv got EOF from 127.0.0.1:51526 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:59:07.467193 23674 catalog_manager.cc:1259] Loaded cluster ID: 7f28d420b49643c4975b96ddc0f4ab5d
I20260927 10:59:07.467267 23674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: loading cluster ID for follower catalog manager: success
I20260927 10:59:07.469034 23674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: acquiring CA information for follower catalog manager: success
I20260927 10:59:07.470323 23674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2409d437379f4ee3be22a7b1bfb0ee2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:59:07.474294 23688 catalog_manager.cc:1259] Loaded cluster ID: 7f28d420b49643c4975b96ddc0f4ab5d
I20260927 10:59:07.474349 23688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: loading cluster ID for follower catalog manager: success
I20260927 10:59:07.475919 23688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: acquiring CA information for follower catalog manager: success
I20260927 10:59:07.477164 23688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 567ce9cf9e974862ae450acd91943ed6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:59:07.680636 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:07.676530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.1.46.253:40227)
0927 10:59:07.676765 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:07.676771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:07.676876 (+ 105us) server_negotiation.cc:407] Connection header received
0927 10:59:07.677103 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:07.677105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:07.677225 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:07.677344 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:07.677611 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.678139 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.679098 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.679330 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.679442 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:07.679775 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:07.680228 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:07.680362 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:07.680426 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 10:59:07.680481 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260927 10:59:07.686998 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:07.683511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.1.46.252:40333)
0927 10:59:07.683724 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:07.683727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:07.683739 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:59:07.683851 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:07.683854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:07.683962 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:07.684059 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:07.684214 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.684709 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.685482 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.685700 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.685779 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:07.686096 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:07.686494 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:07.686642 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:07.686840 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 10:59:07.686886 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260927 10:59:07.693615 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:07.689846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.1.46.253:40227)
0927 10:59:07.689961 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:07.689965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:07.690405 (+ 440us) server_negotiation.cc:407] Connection header received
0927 10:59:07.690511 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:07.690514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:07.690562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:07.690663 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:07.690919 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.691655 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.692504 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.692724 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.692832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:07.693137 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:07.693198 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:07.693274 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:07.693417 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 10:59:07.693465 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:59:07.694281 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:07.689972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.1.46.252:40333)
0927 10:59:07.690090 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:07.690094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:07.690107 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:59:07.690152 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:07.690158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:07.690209 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:07.690295 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:07.690556 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.691259 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.692906 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.693231 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.693321 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:07.693629 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:07.693702 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:07.693763 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:07.694129 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 10:59:07.694175 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:59:07.694983 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:07.689637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.1.46.254:44829)
0927 10:59:07.690407 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:07.690412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:07.690648 (+ 236us) server_negotiation.cc:407] Connection header received
0927 10:59:07.690854 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:07.690856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:07.691035 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:07.691155 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:07.691365 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.692108 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.693310 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:07.693640 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:07.693708 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:07.694163 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:07.694485 (+ 322us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:07.694553 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:07.694763 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 10:59:07.694816 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":78,"threads_started":1}
I20260927 10:59:07.702154 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23435
I20260927 10:59:07.708295 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23510
I20260927 10:59:07.713986 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23584
2026-09-27T10:59:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3031 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:59:07 dist-test-slave-cffh krb5kdc[23801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:59:07 dist-test-slave-cffh krb5kdc[23801](info): set up 2 sockets
Sep 27 10:59:07 dist-test-slave-cffh krb5kdc[23801](info): commencing operation
krb5kdc: starting...
W20260927 10:59:08.739142 1211 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/test-user.keytab.
Sep 27 10:59:08 dist-test-slave-cffh krb5kdc[23801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
Entry for principal kudu/127.1.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab.
I20260927 10:59:08.853835 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40487
--webserver_interface=127.1.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40487,127.1.46.253:42285,127.1.46.252:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:59:08.957401 23814 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:59:08.957703 23814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:08.957782 23814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:08.957875 23814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:08.961853 23814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:08.961933 23814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:08.961974 23814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:08.962028 23814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:08.962067 23814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:08.967061 23814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40487,127.1.46.253:42285,127.1.46.252:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.254.keytab
--principal=kudu/127.1.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.254:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:08.968027 23814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:08.969017 23814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:59:08.976938 23820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:08.977093 23814 server_base.cc:1034] running on GCE node
W20260927 10:59:08.976965 23822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:08.976987 23819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:08.977763 23814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:08.978829 23814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:08.979983 23814 hybrid_clock.cc:648] HybridClock initialized: now 1790506748979967 us; error 33 us; skew 500 ppm
Sep 27 10:59:08 dist-test-slave-cffh krb5kdc[23801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:08.982642 23814 init.cc:377] Logged in from keytab as kudu/127.1.46.254@KRBTEST.COM (short username kudu)
I20260927 10:59:08.983470 23814 webserver.cc:460] Webserver started at http://127.1.46.254:34325/ using document root <none> and password file <none>
I20260927 10:59:08.984014 23814 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:08.984076 23814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:08.984279 23814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:08.986184 23814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data/instance:
uuid: "1219100cbff240db8a7ffdfcfcb01281"
format_stamp: "Formatted at 2026-09-27 10:59:08 on dist-test-slave-cffh"
server_key: "1222aeea5d9b79526ec2e4d19c301505"
server_key_iv: "b38a7c76887301d9739367e543de1714"
server_key_version: "encryptionkey@0"
I20260927 10:59:08.986760 23814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/wal/instance:
uuid: "1219100cbff240db8a7ffdfcfcb01281"
format_stamp: "Formatted at 2026-09-27 10:59:08 on dist-test-slave-cffh"
server_key: "1222aeea5d9b79526ec2e4d19c301505"
server_key_iv: "b38a7c76887301d9739367e543de1714"
server_key_version: "encryptionkey@0"
I20260927 10:59:08.990156 23814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:59:08.992512 23829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:08.993711 23814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:59:08.993853 23814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/wal
uuid: "1219100cbff240db8a7ffdfcfcb01281"
format_stamp: "Formatted at 2026-09-27 10:59:08 on dist-test-slave-cffh"
server_key: "1222aeea5d9b79526ec2e4d19c301505"
server_key_iv: "b38a7c76887301d9739367e543de1714"
server_key_version: "encryptionkey@0"
I20260927 10:59:08.993955 23814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:09.008571 23814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:09.011251 23814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:09.011476 23814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:09.018352 23814 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.254:40487
I20260927 10:59:09.018754 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.254:40487 every 8 connection(s)
I20260927 10:59:09.019276 23814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/data/info.pb
I20260927 10:59:09.021864 23882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:09.025319 23882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:09.029083 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23814
I20260927 10:59:09.029212 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-0/wal/instance
I20260927 10:59:09.029527 1211 external_mini_cluster.cc:1428] Setting key 380884c077b1537844e8cefbb61a3f2f
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.253@KRBTEST.COM" created.
I20260927 10:59:09.039556 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.027245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.1.46.254:40487)
0927 10:59:09.027629 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.027650 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.027803 (+ 153us) server_negotiation.cc:407] Connection header received
0927 10:59:09.028719 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.028723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.028920 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.029097 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.030094 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.031101 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.032047 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.032650 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.032821 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.035080 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.035087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.035098 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.035129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.036768 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.037238 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.037244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.037247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.037392 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.037688 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.037693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.037696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.038041 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.038207 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.038517 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.038708 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"spinlock_wait_cycles":26368,"thread_start_us":150,"threads_started":1}
I20260927 10:59:09.039559 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.027245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40487 (local address 127.0.0.1:60984)
0927 10:59:09.027671 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.027706 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.028460 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.029151 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.029157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.029617 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.029953 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.029972 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.031220 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.031223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.031933 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.031941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.032793 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.032796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.032883 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.033471 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.033494 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.034944 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.036899 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.036909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.036918 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.037133 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.037487 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.037491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.037494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.037599 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.038177 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.038194 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.038303 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.038443 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.038819 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":76,"thread_start_us":103,"threads_started":1}
I20260927 10:59:09.041638 23882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:09.042244 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.041966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34440)
0927 10:59:09.042094 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.042171 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:59:09.042436 23831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
I20260927 10:59:09.043236 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.043004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34454)
0927 10:59:09.043125 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.043176 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:59:09.043380 23882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:42285: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab.
Entry for principal kudu/127.1.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab.
I20260927 10:59:09.053635 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:42285
--webserver_interface=127.1.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab
--principal=kudu/127.1.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40487,127.1.46.253:42285,127.1.46.252:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:59:09.092643 23882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } attempt: 1
I20260927 10:59:09.093557 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.093228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34464)
0927 10:59:09.093401 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.093470 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 10:59:09.094368 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.094128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34472)
0927 10:59:09.094228 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.094295 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:59:09.094537 23882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:42285: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
W20260927 10:59:09.159432 23889 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:59:09.159688 23889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:09.159710 23889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:09.159787 23889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:59:09.162768 23882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } attempt: 2
I20260927 10:59:09.163663 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.163342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34484)
0927 10:59:09.163507 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.163577 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 10:59:09.163899 23889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:09.163964 23889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:09.164000 23889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:09.164055 23889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:09.164093 23889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:09.164467 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.164263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34488)
0927 10:59:09.164369 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.164414 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 10:59:09.164608 23882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.253:42285: Network error: Client connection negotiation failed: client connection to 127.1.46.253:42285: connect: Connection refused (error 111)
I20260927 10:59:09.169345 23889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40487,127.1.46.253:42285,127.1.46.252:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.253.keytab
--principal=kudu/127.1.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.253:42285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:09.170292 23889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:09.171254 23889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:59:09.178885 23895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:09.178906 23894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:09.178894 23897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:09.178997 23889 server_base.cc:1034] running on GCE node
I20260927 10:59:09.179733 23889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:09.180733 23889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:09.181926 23889 hybrid_clock.cc:648] HybridClock initialized: now 1790506749181916 us; error 26 us; skew 500 ppm
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:09.184535 23889 init.cc:377] Logged in from keytab as kudu/127.1.46.253@KRBTEST.COM (short username kudu)
I20260927 10:59:09.185374 23889 webserver.cc:460] Webserver started at http://127.1.46.253:36987/ using document root <none> and password file <none>
I20260927 10:59:09.185883 23889 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:09.185969 23889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:09.186203 23889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:09.188145 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data/instance:
uuid: "c2c294c066f14fde84127ee44da9bcc4"
format_stamp: "Formatted at 2026-09-27 10:59:09 on dist-test-slave-cffh"
server_key: "476f3ceefc8814f9fac50f9723780bab"
server_key_iv: "213fd4bf2854f5725ac07dba5b5b9c68"
server_key_version: "encryptionkey@0"
I20260927 10:59:09.188720 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/wal/instance:
uuid: "c2c294c066f14fde84127ee44da9bcc4"
format_stamp: "Formatted at 2026-09-27 10:59:09 on dist-test-slave-cffh"
server_key: "476f3ceefc8814f9fac50f9723780bab"
server_key_iv: "213fd4bf2854f5725ac07dba5b5b9c68"
server_key_version: "encryptionkey@0"
I20260927 10:59:09.192239 23889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 10:59:09.194500 23904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:09.195657 23889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:59:09.195804 23889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/wal
uuid: "c2c294c066f14fde84127ee44da9bcc4"
format_stamp: "Formatted at 2026-09-27 10:59:09 on dist-test-slave-cffh"
server_key: "476f3ceefc8814f9fac50f9723780bab"
server_key_iv: "213fd4bf2854f5725ac07dba5b5b9c68"
server_key_version: "encryptionkey@0"
I20260927 10:59:09.195917 23889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:09.206251 23889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:09.208840 23889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:09.209033 23889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:09.215549 23889 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.253:42285
I20260927 10:59:09.215847 23956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.253:42285 every 8 connection(s)
I20260927 10:59:09.216512 23889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/data/info.pb
I20260927 10:59:09.218976 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23889
I20260927 10:59:09.219106 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-1/wal/instance
I20260927 10:59:09.219291 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:09.219455 1211 external_mini_cluster.cc:1428] Setting key 6d4516c4d6a23ed3d0ef25bd09522181
I20260927 10:59:09.223178 23957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.46.252@KRBTEST.COM" created.
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:59:09.236892 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.224910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.1.46.254:40487)
0927 10:59:09.225043 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.225046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.225974 (+ 928us) server_negotiation.cc:407] Connection header received
0927 10:59:09.226927 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.226930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.226995 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.227082 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.228180 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.228742 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.230074 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.230293 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.230381 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.232947 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.232953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.232957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.232990 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.234738 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.235273 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.235277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.235281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.235389 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.235734 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.235738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.235739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.235968 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.236073 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.236563 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.236700 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:59:09.237582 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.225308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40487 (local address 127.0.0.1:60986)
0927 10:59:09.225834 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.225871 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.226722 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.227229 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.227235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.227568 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.228021 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.228041 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.228871 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.228873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.229947 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.229961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.230434 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.230441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.230680 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.231337 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.231352 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.232756 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.234873 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.234877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.234887 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.235140 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.235488 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.235491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.235492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.235634 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.236085 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.236090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.236309 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.236464 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.236698 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
I20260927 10:59:09.239120 23957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab.
Entry for principal kudu/127.1.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab.
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:59:09.245321 1211 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZA5khf/build/debug/bin/kudu
/tmp/dist-test-taskZA5khf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:37803
--webserver_interface=127.1.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab
--principal=kudu/127.1.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.46.212:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.46.254:40487,127.1.46.253:42285,127.1.46.252:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:59:09.248803 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.239598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34490)
0927 10:59:09.239760 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.239806 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.239871 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.240413 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.240415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.240592 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.240700 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.240708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.241649 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.241654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.242331 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.242341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.243076 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.243079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.243261 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.243668 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.243680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.245105 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.247198 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.247204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.247208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.247397 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.247717 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.247724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.247726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.247821 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.248246 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.248248 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.248330 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.248468 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.248613 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":57}
I20260927 10:59:09.248833 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.239777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.1.46.253:42285)
0927 10:59:09.240050 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.240059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.240075 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:59:09.240106 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.240110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.240293 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.240436 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.240832 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.241523 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.242460 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.242946 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.243079 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.245234 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.245242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.245252 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.245272 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.247084 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.247513 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.247519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.247522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.247624 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.247908 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.247913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.247917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.248141 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.248295 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.248560 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.248679 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260927 10:59:09.251206 23957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:09.251886 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.251641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44126)
0927 10:59:09.251739 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.251836 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:59:09.252067 23907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
I20260927 10:59:09.252851 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.252629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44140)
0927 10:59:09.252745 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.252805 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 10:59:09.252995 23957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:37803: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
I20260927 10:59:09.255767 23882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } attempt: 3
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:59:09.263258 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.256213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.1.46.253:42285)
0927 10:59:09.256315 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.256321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.256354 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:59:09.256478 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.256482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.256521 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.256595 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.256970 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.257461 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.258234 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.258477 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.258613 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.260474 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.260478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.260481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.260506 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.261736 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.262130 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.262135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.262138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.262205 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.262459 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.262463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.262466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.262686 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.262782 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.263035 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.263128 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:59:09.263269 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.256160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34500)
0927 10:59:09.256289 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.256309 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.256399 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.256611 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.256613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.256772 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.256851 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.256856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.257571 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.257594 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.258106 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.258113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.258612 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.258615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.258719 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.259125 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.259136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.260353 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.261855 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.261859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.261862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.262039 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.262296 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.262299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.262302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.262377 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.262784 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.262787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.262890 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.262989 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.263111 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:59:09.264232 23882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:09.264873 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.264660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44146)
0927 10:59:09.264771 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.264821 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:59:09.265559 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.265323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44152)
0927 10:59:09.265468 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.265513 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 10:59:09.265693 23882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:37803: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
I20260927 10:59:09.296808 23882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } attempt: 1
I20260927 10:59:09.297564 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.297256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44160)
0927 10:59:09.297423 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.297479 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 10:59:09.298296 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.298097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44166)
0927 10:59:09.298199 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.298246 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:59:09.298465 23882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:37803: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
I20260927 10:59:09.302199 23957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } attempt: 1
I20260927 10:59:09.302920 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.302624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44180)
0927 10:59:09.302780 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.302845 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 10:59:09.303598 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.303373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44188)
0927 10:59:09.303486 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.303538 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:59:09.303743 23957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:37803: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
W20260927 10:59:09.354535 23963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:59:09.354797 23963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:59:09.354856 23963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:59:09.354951 23963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:59:09.358958 23963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:59:09.359002 23963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:59:09.359021 23963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:59:09.359061 23963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:59:09.359095 23963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:59:09.363977 23963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.46.212:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.46.254:40487,127.1.46.253:42285,127.1.46.252:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/krb5kdc/kudu_127.1.46.252.keytab
--principal=kudu/127.1.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.46.252:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:59:09.364880 23963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:59:09.365811 23963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:59:09.371940 23957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } attempt: 2
I20260927 10:59:09.373186 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.372443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44202)
0927 10:59:09.372949 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.373005 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409}
I20260927 10:59:09.373607 23882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } attempt: 2
W20260927 10:59:09.373827 23969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:09.374080 23963 server_base.cc:1034] running on GCE node
I20260927 10:59:09.374119 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.373854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44208)
0927 10:59:09.374001 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.374056 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 10:59:09.374188 23970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:59:09.374315 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.373976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44216)
0927 10:59:09.374158 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.374218 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 10:59:09.374338 23972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:59:09.374336 23957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:37803: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
I20260927 10:59:09.374940 23963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:59:09.375136 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.374932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44232)
0927 10:59:09.375032 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.375081 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:59:09.375314 23882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.46.252:37803: Network error: Client connection negotiation failed: client connection to 127.1.46.252:37803: connect: Connection refused (error 111)
I20260927 10:59:09.376026 23963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:59:09.377180 23963 hybrid_clock.cc:648] HybridClock initialized: now 1790506749377169 us; error 21 us; skew 500 ppm
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:59:09.379727 23963 init.cc:377] Logged in from keytab as kudu/127.1.46.252@KRBTEST.COM (short username kudu)
I20260927 10:59:09.380576 23963 webserver.cc:460] Webserver started at http://127.1.46.252:40843/ using document root <none> and password file <none>
I20260927 10:59:09.381093 23963 fs_manager.cc:359] Metadata directory not provided
I20260927 10:59:09.381175 23963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:59:09.381382 23963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:59:09.383267 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data/instance:
uuid: "c5d3298d652643a2a4c44a8552cf6403"
format_stamp: "Formatted at 2026-09-27 10:59:09 on dist-test-slave-cffh"
server_key: "6a2f55c6c804d01c4b03018b25060e67"
server_key_iv: "dd7c8eb1f187b0de0749b7e096f498cd"
server_key_version: "encryptionkey@0"
I20260927 10:59:09.383795 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/wal/instance:
uuid: "c5d3298d652643a2a4c44a8552cf6403"
format_stamp: "Formatted at 2026-09-27 10:59:09 on dist-test-slave-cffh"
server_key: "6a2f55c6c804d01c4b03018b25060e67"
server_key_iv: "dd7c8eb1f187b0de0749b7e096f498cd"
server_key_version: "encryptionkey@0"
I20260927 10:59:09.387104 23963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:59:09.389277 23979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:59:09.390453 23963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:59:09.390625 23963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data,/tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/wal
uuid: "c5d3298d652643a2a4c44a8552cf6403"
format_stamp: "Formatted at 2026-09-27 10:59:09 on dist-test-slave-cffh"
server_key: "6a2f55c6c804d01c4b03018b25060e67"
server_key_iv: "dd7c8eb1f187b0de0749b7e096f498cd"
server_key_version: "encryptionkey@0"
I20260927 10:59:09.390734 23963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:59:09.422540 23963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:59:09.425631 23963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:59:09.425819 23963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:59:09.432415 23963 rpc_server.cc:307] RPC server started. Bound to: 127.1.46.252:37803
I20260927 10:59:09.432763 24031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.46.252:37803 every 8 connection(s)
I20260927 10:59:09.433422 23963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/data/info.pb
I20260927 10:59:09.436055 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:59:09.439481 24032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:09.440989 1211 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZA5khf/build/debug/bin/kudu as pid 23963
I20260927 10:59:09.441123 1211 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZA5khf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790506472751632-1211-0/minicluster-data/master-2/wal/instance
I20260927 10:59:09.441452 1211 external_mini_cluster.cc:1428] Setting key 40057fece22efa3661292ba10f2c244d
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:59:09.453732 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.441162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.1.46.254:40487)
0927 10:59:09.441294 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.441299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.442179 (+ 880us) server_negotiation.cc:407] Connection header received
0927 10:59:09.443257 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.443260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.443308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.443405 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.444422 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.445231 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.446538 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.446899 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.446997 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.449522 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.449525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.449527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.449548 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.451758 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.452287 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.452299 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.452302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.452390 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.452745 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.452747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.452749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.452977 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.453075 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.453438 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.453568 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:59:09.454439 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.442010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.1.46.252:37803)
0927 10:59:09.442415 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.442423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.442554 (+ 131us) server_negotiation.cc:407] Connection header received
0927 10:59:09.443093 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.443111 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.443326 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.443505 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.444286 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.445514 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.446671 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.447169 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.447245 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.449980 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.450001 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.450014 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.450038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.451812 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.452313 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.452319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.452320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.452431 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.452724 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.452728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.452732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.453160 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.453288 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.453565 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.453742 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":314,"thread_start_us":110,"threads_started":1}
I20260927 10:59:09.454437 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.441534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.254:40487 (local address 127.0.0.1:32770)
0927 10:59:09.442008 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.442062 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.443087 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.443470 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.443475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.443944 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.444263 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.444289 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.445338 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.445340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.446423 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.446434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.447039 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.447042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.447172 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.447733 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.447758 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.449381 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.451918 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.451926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.451934 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.452195 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.452498 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.452502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.452504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.452656 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.453076 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.453080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.453186 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.453371 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.453664 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":131,"threads_started":1}
I20260927 10:59:09.455631 24032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.254@KRBTEST.COM
I20260927 10:59:09.464180 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.456064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.253:42285 (local address 127.0.0.1:34514)
0927 10:59:09.456245 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.456256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.456308 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.456790 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.456795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.456938 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.457059 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.457068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.457886 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.457888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.458491 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.458500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.458995 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.458999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.459096 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.459569 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.459582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.460950 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.462832 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.462835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.462836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.463009 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.463287 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.463313 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.463314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.463377 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.463736 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.463739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.463829 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.463971 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.464077 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 10:59:09.464365 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.456397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.1.46.253:42285)
0927 10:59:09.456531 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.456539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.456556 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:59:09.456604 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.456607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.456667 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.456764 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.457201 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.457780 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.458651 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.458882 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.458996 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.461079 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.461082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.461084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.461104 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.462674 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.463103 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.463107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.463109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.463193 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.463457 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.463460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.463462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.463643 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.463743 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.464048 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.464157 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:59:09.464936 24032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:59:09.465431 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.456618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.1.46.254:40487)
0927 10:59:09.456740 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.456746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.456761 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:59:09.456837 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.456840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.456901 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.456981 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.457610 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.458137 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.459020 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.459303 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.459439 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.461893 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.461897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.461901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.461924 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.463449 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.463818 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.463822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.463825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.463934 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.464243 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.464247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.464252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.464619 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.464820 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.465169 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.465283 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:59:09.465494 23957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } attempt: 3
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.252@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506749, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.253@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.46.253@KRBTEST.COM
I20260927 10:59:09.473445 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.465324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44254)
0927 10:59:09.465493 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.465507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.465578 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.465837 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.465840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.465974 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.466025 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.466029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.466940 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.466943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.467715 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.467725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.468343 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.468346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.468392 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.468807 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.468823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.470044 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.472082 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.472088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.472092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.472265 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.472533 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.472539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.472542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.472627 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.472965 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.472969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.473068 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.473182 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.473266 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 10:59:09.473526 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.465384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.1.46.252:37803)
0927 10:59:09.465481 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.465486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.465549 (+ 63us) server_negotiation.cc:407] Connection header received
0927 10:59:09.465662 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.465666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.465705 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.465782 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.466152 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.466842 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.467826 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.468140 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.468227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.470174 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.470179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.470183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.470204 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.472003 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.472353 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.472359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.472362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.472440 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.472737 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.472739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.472740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.472875 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.472970 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.473251 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.473374 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:59:09.474466 23882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } attempt: 3
I20260927 10:59:09.475852 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.465806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44268)
0927 10:59:09.466160 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.466172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.466237 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.466672 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.466675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.466826 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.466926 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.466934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.468279 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.468282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.469044 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.469056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.469487 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.469491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.469574 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.469993 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.470003 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.471475 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.474317 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.474320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.474322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.474643 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.474927 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.474930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.474932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.475012 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.475444 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.475447 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.475526 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.475604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.475719 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260927 10:59:09.476397 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.465926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.1.46.252:37803)
0927 10:59:09.466223 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.466231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.466242 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:59:09.466308 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.466310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.466345 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.467014 (+ 669us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.467078 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.468166 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.469162 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.469383 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.469484 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.471624 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.471627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.471630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.471655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.474209 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.474740 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.474743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.474745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.474819 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.475113 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.475116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.475118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.475316 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.476084 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.476152 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.476248 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260927 10:59:09.476557 24032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: Bootstrap starting.
I20260927 10:59:09.476821 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.466803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.1.46.253:42285)
0927 10:59:09.466888 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.466892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.466960 (+ 68us) server_negotiation.cc:407] Connection header received
0927 10:59:09.467057 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.467060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.467092 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.467175 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.467686 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.468123 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.469125 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.469477 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.469579 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.472466 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.472470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.472473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.472499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.475098 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.475489 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.475493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.475496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.475571 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.475842 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.475847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.475850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.476051 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.476219 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.476592 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.476682 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":14720}
I20260927 10:59:09.478653 23957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: Bootstrap starting.
I20260927 10:59:09.478828 24032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: Neither blocks nor log segments found. Creating new log.
I20260927 10:59:09.479669 24032 log.cc:824] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:09.480990 23957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: Neither blocks nor log segments found. Creating new log.
I20260927 10:59:09.481760 1211 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:59:09.481839 23957 log.cc:824] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:09.483664 24032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: No bootstrap required, opened a new log
I20260927 10:59:09.483755 23957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: No bootstrap required, opened a new log
Sep 27 10:59:09 dist-test-slave-cffh krb5kdc[23801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.46.254@KRBTEST.COM for kudu/127.1.46.252@KRBTEST.COM
I20260927 10:59:09.482929 23856 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1219100cbff240db8a7ffdfcfcb01281"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32782
I20260927 10:59:09.487334 23957 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.487572 23957 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c294c066f14fde84127ee44da9bcc4, State: Initialized, Role: FOLLOWER
I20260927 10:59:09.487848 23957 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.488269 24032 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.488493 24032 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d3298d652643a2a4c44a8552cf6403, State: Initialized, Role: FOLLOWER
I20260927 10:59:09.488415 24040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.488703 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.474951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.46.252:37803 (local address 127.0.0.1:44270)
0927 10:59:09.475035 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 10:59:09.475050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:59:09.475114 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:59:09.479444 (+ 4330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:59:09.479447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:59:09.479622 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:59:09.479722 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.479729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.481112 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.481117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.481872 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:59:09.481882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.482485 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.482488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:59:09.482613 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.483012 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:59:09.483025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:59:09.484397 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:59:09.487027 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.487031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.487033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.487266 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.487705 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:59:09.487708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:59:09.487710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:59:09.487817 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:59:09.488246 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:59:09.488249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:59:09.488343 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 10:59:09.488420 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:59:09.488549 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 10:59:09.488821 24032 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.489027 24040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:09.489208 23957 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:09.489228 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.478863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.1.46.252:37803)
0927 10:59:09.479222 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.479226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.479239 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:59:09.479269 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.479271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.479306 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.480228 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.480308 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.480991 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.482005 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.482375 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.482463 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.484574 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.484578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.484580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.484626 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.486901 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.487400 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.487403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.487405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.487490 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.487925 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.487928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.487930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.488131 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.488947 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.489014 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.489110 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260927 10:59:09.489989 24032 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:09.492031 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281: Bootstrap starting.
I20260927 10:59:09.494417 23882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281: Neither blocks nor log segments found. Creating new log.
W20260927 10:59:09.495042 24063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:59:09.495103 24063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:59:09.495285 23882 log.cc:824] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281: Log is configured to *not* fsync() on all Append() calls
I20260927 10:59:09.495343 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.495440 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:09.493456 23889 master_runner.cc:186] Error getting master registration for 127.1.46.254:40487: OK, instance_id { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" instance_seqno: 1790506749016450 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:59:09.497113 23889 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:59:09.497277 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281: No bootstrap required, opened a new log
I20260927 10:59:09.493992 23963 master_runner.cc:186] Error getting master registration for 127.1.46.254:40487: OK, instance_id { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" instance_seqno: 1790506749016450 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:59:09.498035 23963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:59:09.499594 24057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:59:09.499655 24057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:59:09.499930 23882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.500149 23882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1219100cbff240db8a7ffdfcfcb01281, State: Initialized, Role: FOLLOWER
I20260927 10:59:09.500476 23882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.500949 24067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.501082 24067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:09.501669 23882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:59:09.501770 23856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:59:09.501875 23856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:09.503005 23856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.504423 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1219100cbff240db8a7ffdfcfcb01281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2c294c066f14fde84127ee44da9bcc4"
I20260927 10:59:09.504483 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1219100cbff240db8a7ffdfcfcb01281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c5d3298d652643a2a4c44a8552cf6403"
I20260927 10:59:09.504635 23931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:09.504705 24006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:59:09.504002 23856 leader_election.cc:288] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [CANDIDATE]: Term 1 election: Requested vote from peers c2c294c066f14fde84127ee44da9bcc4 (127.1.46.253:42285), c5d3298d652643a2a4c44a8552cf6403 (127.1.46.252:37803)
I20260927 10:59:09.505901 23931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1219100cbff240db8a7ffdfcfcb01281 in term 1.
I20260927 10:59:09.505956 24006 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1219100cbff240db8a7ffdfcfcb01281 in term 1.
I20260927 10:59:09.506760 23832 leader_election.cc:302] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1219100cbff240db8a7ffdfcfcb01281, c5d3298d652643a2a4c44a8552cf6403; no voters:
I20260927 10:59:09.507778 24067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:59:09.507979 24067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [term 1 LEADER]: Becoming Leader. State: Replica: 1219100cbff240db8a7ffdfcfcb01281, State: Running, Role: LEADER
I20260927 10:59:09.508270 24067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } }
I20260927 10:59:09.510401 24084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1219100cbff240db8a7ffdfcfcb01281. Latest consensus state: current_term: 1 leader_uuid: "1219100cbff240db8a7ffdfcfcb01281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.510519 24084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: This master's current role is: LEADER
W20260927 10:59:09.511411 24078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:59:09.511483 24078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:59:09.511550 23814 master_runner.cc:263] No leader master found from master 1219100cbff240db8a7ffdfcfcb01281
I20260927 10:59:09.511605 23814 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:59:09.512293 24086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 10:59:09.512816 23835 tablet.cc:2369] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:59:09.513244 24086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:59:09.519634 24006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [term 1 FOLLOWER]: Refusing update from remote peer 1219100cbff240db8a7ffdfcfcb01281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:59:09.520287 24084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:59:09.522342 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.511897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.1.46.252:37803)
0927 10:59:09.512025 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.512030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.512203 (+ 173us) server_negotiation.cc:407] Connection header received
0927 10:59:09.512348 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.512351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.512399 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.512483 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.513437 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.514233 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.515422 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.515756 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.515832 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.517599 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.517603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.517605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.517629 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.519468 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.519928 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.519931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.519935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.520015 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.521388 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.521392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.521393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.521601 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.521730 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.522085 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.522194 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:59:09.524109 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1219100cbff240db8a7ffdfcfcb01281. Latest consensus state: current_term: 1 leader_uuid: "1219100cbff240db8a7ffdfcfcb01281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.524252 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:09.526818 23931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [term 1 FOLLOWER]: Refusing update from remote peer 1219100cbff240db8a7ffdfcfcb01281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:59:09.527441 24084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:59:09.531865 24040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1219100cbff240db8a7ffdfcfcb01281. Latest consensus state: current_term: 1 leader_uuid: "1219100cbff240db8a7ffdfcfcb01281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.531982 24040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:09.532703 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.515825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.1.46.254:40487)
0927 10:59:09.515947 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.515951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.522937 (+ 6986us) server_negotiation.cc:407] Connection header received
0927 10:59:09.523096 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.523101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.523151 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.523199 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.524055 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.524763 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.525893 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.526196 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.526264 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.528053 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.528056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.528059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.528084 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.529722 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.530291 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.530295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.530297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.530497 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.531010 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.531014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.531016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.531236 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.531355 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.532448 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.532555 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:59:09.538285 24067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1219100cbff240db8a7ffdfcfcb01281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.538394 24067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:09.538503 24088 mvcc.cc:205] Tried to move back new op lower bound from 7333915646024605696 to 7333915645989105664. Current Snapshot: MvccSnapshot[applied={T|T < 7333915646024605696}]
I20260927 10:59:09.539024 24040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1219100cbff240db8a7ffdfcfcb01281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.539139 24040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:09.539450 24093 mvcc.cc:205] Tried to move back new op lower bound from 7333915646024605696 to 7333915645989105664. Current Snapshot: MvccSnapshot[applied={T|T < 7333915646024605696}]
I20260927 10:59:09.539943 24086 catalog_manager.cc:1347] Generated new cluster ID: 4387277162824263900b269b1cf28a8d
I20260927 10:59:09.540001 24086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:59:09.546849 24084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1219100cbff240db8a7ffdfcfcb01281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.546972 24084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281 [sys.catalog]: This master's current role is: LEADER
I20260927 10:59:09.547989 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1219100cbff240db8a7ffdfcfcb01281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1219100cbff240db8a7ffdfcfcb01281" member_type: VOTER last_known_addr { host: "127.1.46.254" port: 40487 } } peers { permanent_uuid: "c2c294c066f14fde84127ee44da9bcc4" member_type: VOTER last_known_addr { host: "127.1.46.253" port: 42285 } } peers { permanent_uuid: "c5d3298d652643a2a4c44a8552cf6403" member_type: VOTER last_known_addr { host: "127.1.46.252" port: 37803 } } }
I20260927 10:59:09.548110 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:59:09.548697 24091 mvcc.cc:205] Tried to move back new op lower bound from 7333915646024605696 to 7333915645989105664. Current Snapshot: MvccSnapshot[applied={T|T < 7333915646024605696}]
I20260927 10:59:09.555680 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:09.515219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.1.46.253:42285)
0927 10:59:09.515326 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.515332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.518475 (+ 3143us) server_negotiation.cc:407] Connection header received
0927 10:59:09.518617 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.518620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.518672 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.518740 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.531447 (+ 12707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.532213 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.538662 (+ 6449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:09.538965 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:09.539038 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:09.543498 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:59:09.543503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:59:09.543506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:59:09.543531 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:59:09.545105 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.549219 (+ 4114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.549223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.549225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.549308 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:59:09.554681 (+ 5373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:59:09.554685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:59:09.554687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:59:09.554916 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:59:09.555059 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:09.555398 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 10:59:09.555522 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:59:09.584903 24086 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:59:09.586205 24086 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:59:09.635123 24086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1219100cbff240db8a7ffdfcfcb01281: Generated new TSK 0
I20260927 10:59:09.636039 24086 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:59:09.663487 24037 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.660472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.1.46.252:37803)
0927 10:59:09.660594 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.660599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.662392 (+ 1793us) server_negotiation.cc:407] Connection header received
0927 10:59:09.662621 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.662625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.662677 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.662732 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.663355 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44300: BlockingRecv error: recv got EOF from 127.0.0.1:44300 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 10:59:09.665028 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.661148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.1.46.253:42285)
0927 10:59:09.661255 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.661259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.663952 (+ 2693us) server_negotiation.cc:407] Connection header received
0927 10:59:09.664370 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.664374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.664424 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.664472 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.664924 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34554: BlockingRecv error: recv got EOF from 127.0.0.1:34554 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 10:59:09.666394 23885 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.661811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.1.46.254:40487)
0927 10:59:09.661919 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.661923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.665505 (+ 3582us) server_negotiation.cc:407] Connection header received
0927 10:59:09.665732 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.665735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.665782 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.665831 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.666275 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32792: BlockingRecv error: recv got EOF from 127.0.0.1:32792 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 10:59:09.670442 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.669029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.1.46.253:42285)
0927 10:59:09.669122 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.669125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.669527 (+ 402us) server_negotiation.cc:407] Connection header received
0927 10:59:09.669714 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.669717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.669783 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.669836 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.670348 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34556: BlockingRecv error: recv got EOF from 127.0.0.1:34556 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 10:59:09.675177 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.673777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.1.46.253:42285)
0927 10:59:09.673884 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.673888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.674352 (+ 464us) server_negotiation.cc:407] Connection header received
0927 10:59:09.674523 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.674526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.674614 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.674664 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.675099 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34572: BlockingRecv error: recv got EOF from 127.0.0.1:34572 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 10:59:09.680536 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.679148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.1.46.253:42285)
0927 10:59:09.679241 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.679244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.679726 (+ 482us) server_negotiation.cc:407] Connection header received
0927 10:59:09.679903 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.679905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.679952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.679998 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.680454 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34582: BlockingRecv error: recv got EOF from 127.0.0.1:34582 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 10:59:09.687994 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.686502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.1.46.253:42285)
0927 10:59:09.686637 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.686641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.687088 (+ 447us) server_negotiation.cc:407] Connection header received
0927 10:59:09.687272 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.687277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.687355 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.687415 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.687894 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34588: BlockingRecv error: recv got EOF from 127.0.0.1:34588 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 10:59:09.700237 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.698330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.1.46.253:42285)
0927 10:59:09.698436 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.698440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.699119 (+ 679us) server_negotiation.cc:407] Connection header received
0927 10:59:09.699397 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.699400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.699456 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.699512 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.700136 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34594: BlockingRecv error: recv got EOF from 127.0.0.1:34594 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 10:59:09.720218 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.718630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.1.46.253:42285)
0927 10:59:09.718738 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.718742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.719275 (+ 533us) server_negotiation.cc:407] Connection header received
0927 10:59:09.719478 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.719482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.719532 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.719583 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.720108 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34600: BlockingRecv error: recv got EOF from 127.0.0.1:34600 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 10:59:09.755676 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.754680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.1.46.253:42285)
0927 10:59:09.754824 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.754828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.754927 (+ 99us) server_negotiation.cc:407] Connection header received
0927 10:59:09.755047 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.755049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.755097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.755180 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.755577 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34610: BlockingRecv error: recv got EOF from 127.0.0.1:34610 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:59:09.823089 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.821914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.1.46.253:42285)
0927 10:59:09.822066 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.822070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.822201 (+ 131us) server_negotiation.cc:407] Connection header received
0927 10:59:09.822331 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.822333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.822383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.822470 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.822990 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34620: BlockingRecv error: recv got EOF from 127.0.0.1:34620 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 10:59:09.954339 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:09.953279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.1.46.253:42285)
0927 10:59:09.953438 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:09.953442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:09.953550 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:59:09.953669 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:09.953672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:09.953722 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:09.953810 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:09.954219 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34630: BlockingRecv error: recv got EOF from 127.0.0.1:34630 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 10:59:10.213609 23962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:59:10.212620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.1.46.253:42285)
0927 10:59:10.212764 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:10.212768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:10.212879 (+ 111us) server_negotiation.cc:407] Connection header received
0927 10:59:10.212999 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:10.213001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:10.213049 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:10.213126 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:59:10.213514 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34632: BlockingRecv error: recv got EOF from 127.0.0.1:34632 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:59:10.496317 24063 catalog_manager.cc:1259] Loaded cluster ID: 4387277162824263900b269b1cf28a8d
I20260927 10:59:10.496421 24063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: loading cluster ID for follower catalog manager: success
I20260927 10:59:10.498281 24063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: acquiring CA information for follower catalog manager: success
I20260927 10:59:10.499532 24063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5d3298d652643a2a4c44a8552cf6403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:59:10.500630 24057 catalog_manager.cc:1259] Loaded cluster ID: 4387277162824263900b269b1cf28a8d
I20260927 10:59:10.500699 24057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: loading cluster ID for follower catalog manager: success
I20260927 10:59:10.502377 24057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: acquiring CA information for follower catalog manager: success
I20260927 10:59:10.503583 24057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2c294c066f14fde84127ee44da9bcc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:59:10.731237 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:10.727660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.1.46.253:42285)
0927 10:59:10.727893 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:10.727897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:10.727991 (+ 94us) server_negotiation.cc:407] Connection header received
0927 10:59:10.728106 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:10.728108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:10.728230 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:10.728335 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:10.728556 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.729089 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.729897 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.730109 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.730184 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:10.730421 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:10.730935 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:10.731029 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:10.731079 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 10:59:10.731122 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260927 10:59:10.737675 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:10.734011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.1.46.252:37803)
0927 10:59:10.734268 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:10.734271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:10.734284 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:59:10.734387 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:10.734389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:10.734497 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:10.734650 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:10.734803 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.735280 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.736306 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.736539 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.736618 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:10.736872 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:10.737213 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:10.737299 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:10.737430 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 10:59:10.737475 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
I20260927 10:59:10.744592 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:10.740407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.1.46.253:42285)
0927 10:59:10.740508 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:10.740512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:10.740865 (+ 353us) server_negotiation.cc:407] Connection header received
0927 10:59:10.740959 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:10.740962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:10.741009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:10.741080 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:10.741321 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.742191 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.743215 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.743521 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.743630 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:10.744067 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:10.744153 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:10.744200 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:10.744386 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 10:59:10.744443 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:59:10.745263 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:10.740407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.1.46.252:37803)
0927 10:59:10.740635 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:10.740640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:10.741506 (+ 866us) server_negotiation.cc:407] Connection header received
0927 10:59:10.741611 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:10.741615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:10.741662 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:10.741747 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:10.742006 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.742685 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.743951 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.744305 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.744380 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:10.744755 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:10.744845 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:10.744896 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:10.745099 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:59:10.745154 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260927 10:59:10.745591 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:59:10.740400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.1.46.254:40487)
0927 10:59:10.740864 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 10:59:10.740878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:59:10.740892 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:59:10.740973 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:59:10.740975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:59:10.741120 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:59:10.741228 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:59:10.741460 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.742371 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.743744 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:59:10.744117 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:59:10.744192 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:59:10.744677 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:59:10.744996 (+ 319us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:59:10.745064 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:59:10.745249 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 10:59:10.745340 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":64,"threads_started":1}
I20260927 10:59:10.748768 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23814
I20260927 10:59:10.754772 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23889
I20260927 10:59:10.760834 1211 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZA5khf/build/debug/bin/kudu with pid 23963
2026-09-27T10:59:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3046 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6078 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278003 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST